2025-12-04T12:25:50.4591798Z Current runner version: '2.329.0'
2025-12-04T12:25:50.4594945Z Runner name: 'linux.rocm.gpu.gfx942.4.b-bphpw-runner-bfc27'
2025-12-04T12:25:50.4595345Z Runner group name: 'default'
2025-12-04T12:25:50.4595777Z Machine name: 'linux'
2025-12-04T12:25:50.4596946Z ##[group]GITHUB_TOKEN Permissions
2025-12-04T12:25:50.4598102Z Contents: read
2025-12-04T12:25:50.4598352Z Metadata: read
2025-12-04T12:25:50.4598596Z ##[endgroup]
2025-12-04T12:25:50.4599662Z Secret source: Actions
2025-12-04T12:25:50.4599964Z Prepare workflow directory
2025-12-04T12:25:50.4842960Z Prepare all required actions
2025-12-04T12:25:50.4862950Z Getting action download info
2025-12-04T12:25:50.9323768Z Download action repository 'pytorch/pytorch@main' (SHA:685ba6bc01170c0cb793f872e56164df89b63573)
2025-12-04T12:25:54.8134027Z Download action repository 'pytorch/test-infra@main' (SHA:39aa74d619174326f4e2fb0e216151c2f29d9ffd)
2025-12-04T12:25:55.9778444Z Download action repository 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-12-04T12:25:56.8890238Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722)
2025-12-04T12:25:57.7240812Z Getting action download info
2025-12-04T12:25:57.9297305Z Download action repository 'actions/checkout@v4' (SHA:34e114876b0b11c390a56381ad16ebd13914f8d5)
2025-12-04T12:25:58.8560375Z Getting action download info
2025-12-04T12:25:59.0784172Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2025-12-04T12:25:59.7930194Z Getting action download info
2025-12-04T12:26:00.0147525Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/main (ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32)
2025-12-04T12:26:00.0149470Z ##[group] Inputs
2025-12-04T12:26:00.0149625Z   build-environment: linux-noble-rocm-py3.12-mi300
2025-12-04T12:26:00.0150947Z   test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}]}
2025-12-04T12:26:00.0152478Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:00.0152765Z   sync-tag: 
2025-12-04T12:26:00.0153135Z   timeout-minutes: 300
2025-12-04T12:26:00.0153240Z   tests-to-include: 
2025-12-04T12:26:00.0153339Z   dashboard-tag: 
2025-12-04T12:26:00.0153559Z   disable-monitor: true
2025-12-04T12:26:00.0153673Z   monitor-log-interval: 5
2025-12-04T12:26:00.0153791Z   monitor-data-collect-interval: 1
2025-12-04T12:26:00.0153916Z ##[endgroup]
2025-12-04T12:26:00.0154154Z Complete job name: linux-noble-rocm-py3.12-mi300 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4.b, module:rocm, oncall:distributed, mem_leak_check)
2025-12-04T12:26:00.0418593Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2025-12-04T12:26:00.0418874Z with:
2025-12-04T12:26:00.0418966Z   no-sudo: true
2025-12-04T12:26:00.0419225Z   submodules: recursive
2025-12-04T12:26:00.0419332Z   fetch-depth: 0
2025-12-04T12:26:00.0419474Z env:
2025-12-04T12:26:00.0419576Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:00.0419689Z ##[endgroup]
2025-12-04T12:26:00.0461515Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-12-04T12:26:00.0461887Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-12-04T12:26:00.0468580Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:00.0468732Z env:
2025-12-04T12:26:00.0468826Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:00.0468926Z ##[endgroup]
2025-12-04T12:26:00.0628218Z ##[group]Run actions/checkout@v4
2025-12-04T12:26:00.0628406Z with:
2025-12-04T12:26:00.0628531Z   ref: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:00.0628670Z   fetch-depth: 0
2025-12-04T12:26:00.0628769Z   submodules: recursive
2025-12-04T12:26:00.0628898Z   show-progress: false
2025-12-04T12:26:00.0629006Z   repository: pytorch/pytorch
2025-12-04T12:26:00.0629188Z   token: ***
2025-12-04T12:26:00.0629289Z   ssh-strict: true
2025-12-04T12:26:00.0629382Z   ssh-user: git
2025-12-04T12:26:00.0629486Z   persist-credentials: true
2025-12-04T12:26:00.0629596Z   clean: true
2025-12-04T12:26:00.0629697Z   sparse-checkout-cone-mode: true
2025-12-04T12:26:00.0629816Z   fetch-tags: false
2025-12-04T12:26:00.0629905Z   lfs: false
2025-12-04T12:26:00.0630000Z   set-safe-directory: true
2025-12-04T12:26:00.0630107Z env:
2025-12-04T12:26:00.0630194Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:00.0630302Z ##[endgroup]
2025-12-04T12:26:00.1185638Z Syncing repository: pytorch/pytorch
2025-12-04T12:26:00.1186264Z ##[group]Getting Git version info
2025-12-04T12:26:00.1186434Z Working directory is '/home/runner/_work/pytorch/pytorch'
2025-12-04T12:26:00.1186686Z [command]/usr/bin/git version
2025-12-04T12:26:00.1186852Z git version 2.52.0
2025-12-04T12:26:00.1193333Z ##[endgroup]
2025-12-04T12:26:00.1199077Z Copying '/home/runner/.gitconfig' to '/home/runner/_work/_temp/90cc88ee-394e-4c9d-be54-5067cd536bf6/.gitconfig'
2025-12-04T12:26:00.1205397Z Temporarily overriding HOME='/home/runner/_work/_temp/90cc88ee-394e-4c9d-be54-5067cd536bf6' before making global git config changes
2025-12-04T12:26:00.1205722Z Adding repository directory to the temporary git global config as a safe directory
2025-12-04T12:26:00.1208409Z [command]/usr/bin/git config --global --add safe.directory /home/runner/_work/pytorch/pytorch
2025-12-04T12:26:00.1232637Z [command]/usr/bin/git config --local --get remote.origin.url
2025-12-04T12:26:00.1252924Z https://github.com/pytorch/pytorch
2025-12-04T12:26:00.1265525Z ##[group]Removing previously created refs, to avoid conflicts
2025-12-04T12:26:00.1268708Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2025-12-04T12:26:00.1282959Z refs/heads/main
2025-12-04T12:26:00.1289673Z [command]/usr/bin/git checkout --detach
2025-12-04T12:26:01.7480319Z HEAD is now at 685ba6bc0117 add back legalize_graph for BC reason (#169541)
2025-12-04T12:26:01.7552173Z [command]/usr/bin/git branch --delete --force main
2025-12-04T12:26:01.7724644Z Deleted branch main (was 685ba6bc0117).
2025-12-04T12:26:01.7729960Z ##[endgroup]
2025-12-04T12:26:01.7735531Z [command]/usr/bin/git submodule status
2025-12-04T12:26:01.7964162Z  7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe)
2025-12-04T12:26:01.8021432Z  4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081)
2025-12-04T12:26:01.8075246Z  b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327)
2025-12-04T12:26:01.8144583Z  c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0)
2025-12-04T12:26:01.8185355Z  3ebbc93ded7285963bff932c678fa367eb393ba6 third_party/NVTX (v3.1.0-313-g3ebbc93)
2025-12-04T12:26:01.8242832Z  1d8f600fd424278486eade7ed3e877c99f0846b1 third_party/VulkanMemoryAllocator (v2.1.0-982-g1d8f600)
2025-12-04T12:26:01.8567191Z  51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a0103656)
2025-12-04T12:26:01.8591907Z  01aae101b9e5e94d6c16a9514c9fb8df99c93150 third_party/aiter (v0.1.1-92-g01aae101)
2025-12-04T12:26:01.8609678Z  299e5928955cc62af9968370293b916f5130916f third_party/benchmark (v1.9.3)
2025-12-04T12:26:01.8672749Z  7fe50dc3da2069d6645d9deb8c017a876472a977 third_party/composable_kernel (rocm-6.4.3-459-g7fe50dc3d)
2025-12-04T12:26:01.8765151Z  89c932f313c6437c38f2982869beacc89c2f2246 third_party/cpp-httplib (v0.26.0)
2025-12-04T12:26:01.8845213Z  f858c30bcb16f8effd5ff46996f0514539e17abc third_party/cpuinfo (f858c30)
2025-12-04T12:26:01.8874852Z  0b1577c8c83401237d601d0d0db5210506705396 third_party/cudnn_frontend (v0.5-61-g0b1577c)
2025-12-04T12:26:01.8943114Z  f88806b1e31dfa579842638740216dd41fc6c588 third_party/cutlass (v4.3.1)
2025-12-04T12:26:01.8976680Z  c0b988d39a9e47c794d699f29930ed4d7c7e13a4 third_party/fbgemm (v1.4.0-rc1-2-gc0b988d39)
2025-12-04T12:26:01.9033208Z  979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4)
2025-12-04T12:26:01.9050289Z  a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757 third_party/flatbuffers (v24.12.23)
2025-12-04T12:26:01.9285062Z  407c905e45ad75fc29bf0f9bb7c5c2fd3475976f third_party/fmt (12.1.0)
2025-12-04T12:26:01.9352944Z  3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17)
2025-12-04T12:26:01.9429815Z  54cbae0d3a67fa890b4c3d9ee162b7860315e341 third_party/gloo (remotes/origin/gh/c-p-i-o/1/base-37-g54cbae0)
2025-12-04T12:26:01.9577575Z  52eb8108c5bdec04579160ae17225d66034bd723 third_party/googletest (release-1.8.0-3544-g52eb8108)
2025-12-04T12:26:01.9624171Z  719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1)
2025-12-04T12:26:01.9666294Z  dec1d23ca65ab069d225dfe40dea14f455170959 third_party/ittapi (v3.25.5)
2025-12-04T12:26:01.9800020Z  31f85df8fbd89c188f14ef10f1ec65379786b943 third_party/kineto (heads/main)
2025-12-04T12:26:01.9820765Z  d7770c89632329a9914ef1a90289917597639cbe third_party/kleidiai (v1.15.0)
2025-12-04T12:26:01.9835312Z  fbd8b99c2b828428947d70fdc046bb55609be93e third_party/mimalloc (v2.2.4)
2025-12-04T12:26:01.9860117Z  55f93686c01528224f448c19128836e7df245f72 third_party/nlohmann (v3.12.0)
2025-12-04T12:26:02.0064482Z  e709452ef2bbc1d113faf678c24e6d3467696e83 third_party/onnx (v1.18.0)
2025-12-04T12:26:02.0082971Z  a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2)
2025-12-04T12:26:02.0112400Z  0fa0ef591e38c2758e3184c6c23e497b9f732ffa third_party/pocketfft (release_for_eigen-40-g0fa0ef5)
2025-12-04T12:26:02.0338702Z  d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b)
2025-12-04T12:26:02.0399888Z  072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master)
2025-12-04T12:26:02.0448860Z  4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e)
2025-12-04T12:26:02.0468209Z  f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8 third_party/pybind11 (v3.0.1)
2025-12-04T12:26:02.0518223Z  f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated)
2025-12-04T12:26:02.0570178Z  5a1d179df9cf652951b59010a2d2075372d67f68 third_party/sleef (3.8)
2025-12-04T12:26:02.0617853Z  2b4cd91092d335a697416b2a3cb398283246849d third_party/tensorpipe (heads/main)
2025-12-04T12:26:02.0631288Z ##[group]Cleaning the repository
2025-12-04T12:26:02.0636675Z [command]/usr/bin/git clean -ffdx
2025-12-04T12:26:02.0771395Z [command]/usr/bin/git reset --hard HEAD
2025-12-04T12:26:02.1635209Z HEAD is now at 685ba6bc0117 add back legalize_graph for BC reason (#169541)
2025-12-04T12:26:02.1705472Z ##[endgroup]
2025-12-04T12:26:02.1708297Z ##[group]Disabling automatic garbage collection
2025-12-04T12:26:02.1723558Z [command]/usr/bin/git config --local gc.auto 0
2025-12-04T12:26:02.1748783Z ##[endgroup]
2025-12-04T12:26:02.1749105Z ##[group]Setting up auth
2025-12-04T12:26:02.1752025Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-12-04T12:26:02.1775986Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-12-04T12:26:02.2005152Z Entering 'android/libs/fbjni'
2025-12-04T12:26:02.2044725Z Entering 'third_party/FP16'
2025-12-04T12:26:02.2083095Z Entering 'third_party/FXdiv'
2025-12-04T12:26:02.2117217Z Entering 'third_party/NNPACK'
2025-12-04T12:26:02.2154079Z Entering 'third_party/NVTX'
2025-12-04T12:26:02.2186400Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:02.2210672Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:02.2246627Z Entering 'third_party/aiter'
2025-12-04T12:26:02.2276879Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:02.2313350Z Entering 'third_party/benchmark'
2025-12-04T12:26:02.2339133Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:02.2379179Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:02.2412980Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:02.2441717Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:02.2468414Z Entering 'third_party/cutlass'
2025-12-04T12:26:02.2503215Z Entering 'third_party/fbgemm'
2025-12-04T12:26:02.2534435Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:02.2562735Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:02.2594829Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:02.2621236Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:02.2650604Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:02.2681331Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:02.2707840Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:02.2742578Z Entering 'third_party/flash-attention'
2025-12-04T12:26:02.2771959Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:02.2806246Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:02.2840782Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:02.2877382Z Entering 'third_party/fmt'
2025-12-04T12:26:02.2904221Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:02.2927696Z Entering 'third_party/gloo'
2025-12-04T12:26:02.2953186Z Entering 'third_party/googletest'
2025-12-04T12:26:02.2981217Z Entering 'third_party/ideep'
2025-12-04T12:26:02.3002583Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:02.3037775Z Entering 'third_party/ittapi'
2025-12-04T12:26:02.3068375Z Entering 'third_party/kineto'
2025-12-04T12:26:02.3094772Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:02.3121828Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:02.3147869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:02.3174893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:02.3207563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:02.3231751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:02.3269817Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:02.3296450Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:02.3321559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:02.3348457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:02.3370670Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:02.3401750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:02.3426977Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:02.3454709Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:02.3479739Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:02.3507854Z Entering 'third_party/kleidiai'
2025-12-04T12:26:02.3532420Z Entering 'third_party/mimalloc'
2025-12-04T12:26:02.3561163Z Entering 'third_party/nlohmann'
2025-12-04T12:26:02.3586861Z Entering 'third_party/onnx'
2025-12-04T12:26:02.3620834Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:02.3652537Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:02.3681682Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:02.3703318Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:02.3726108Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:02.3753844Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:02.3778010Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:02.3807157Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:02.3837152Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:02.3863706Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:02.3888870Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:02.3914783Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:02.3949074Z Entering 'third_party/pocketfft'
2025-12-04T12:26:02.3982276Z Entering 'third_party/protobuf'
2025-12-04T12:26:02.4012901Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:02.4040772Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:02.4068189Z Entering 'third_party/psimd'
2025-12-04T12:26:02.4094583Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:02.4117169Z Entering 'third_party/pybind11'
2025-12-04T12:26:02.4147975Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:02.4177843Z Entering 'third_party/sleef'
2025-12-04T12:26:02.4209504Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:02.4236331Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:02.4259163Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:02.4283572Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:02.4306823Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:02.4334301Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:02.4378397Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-12-04T12:26:02.4397058Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-12-04T12:26:02.4574562Z Entering 'android/libs/fbjni'
2025-12-04T12:26:02.4596690Z Entering 'third_party/FP16'
2025-12-04T12:26:02.4619758Z Entering 'third_party/FXdiv'
2025-12-04T12:26:02.4642798Z Entering 'third_party/NNPACK'
2025-12-04T12:26:02.4663866Z Entering 'third_party/NVTX'
2025-12-04T12:26:02.4686967Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:02.4709727Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:02.4737085Z Entering 'third_party/aiter'
2025-12-04T12:26:02.4760292Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:02.4790479Z Entering 'third_party/benchmark'
2025-12-04T12:26:02.4813990Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:02.4841922Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:02.4865524Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:02.4892454Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:02.4914895Z Entering 'third_party/cutlass'
2025-12-04T12:26:02.4939774Z Entering 'third_party/fbgemm'
2025-12-04T12:26:02.4968036Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:02.4989558Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:02.5014060Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:02.5034577Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:02.5059239Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:02.5079378Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:02.5100242Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:02.5123665Z Entering 'third_party/flash-attention'
2025-12-04T12:26:02.5146396Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:02.5173010Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:02.5198817Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:02.5221821Z Entering 'third_party/fmt'
2025-12-04T12:26:02.5248406Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:02.5273539Z Entering 'third_party/gloo'
2025-12-04T12:26:02.5298903Z Entering 'third_party/googletest'
2025-12-04T12:26:02.5319473Z Entering 'third_party/ideep'
2025-12-04T12:26:02.5342058Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:02.5374898Z Entering 'third_party/ittapi'
2025-12-04T12:26:02.5402163Z Entering 'third_party/kineto'
2025-12-04T12:26:02.5429089Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:02.5451248Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:02.5471968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:02.5495908Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:02.5518029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:02.5537072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:02.5559203Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:02.5580029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:02.5618139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:02.5646602Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:02.5668850Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:02.5692464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:02.5728197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:02.5756472Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:02.5785319Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:02.5827670Z Entering 'third_party/kleidiai'
2025-12-04T12:26:02.5852620Z Entering 'third_party/mimalloc'
2025-12-04T12:26:02.5884577Z Entering 'third_party/nlohmann'
2025-12-04T12:26:02.5908506Z Entering 'third_party/onnx'
2025-12-04T12:26:02.5936382Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:02.5981133Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:02.6008179Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:02.6028775Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:02.6050777Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:02.6071947Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:02.6093151Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:02.6118332Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:02.6145008Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:02.6164444Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:02.6188496Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:02.6221055Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:02.6249326Z Entering 'third_party/pocketfft'
2025-12-04T12:26:02.6282154Z Entering 'third_party/protobuf'
2025-12-04T12:26:02.6310442Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:02.6332094Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:02.6359756Z Entering 'third_party/psimd'
2025-12-04T12:26:02.6385237Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:02.6414085Z Entering 'third_party/pybind11'
2025-12-04T12:26:02.6445272Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:02.6477114Z Entering 'third_party/sleef'
2025-12-04T12:26:02.6500040Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:02.6526000Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:02.6547315Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:02.6570836Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:02.6601004Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:02.6627888Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:02.6673363Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.6694003Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
2025-12-04T12:26:02.6897189Z Entering 'android/libs/fbjni'
2025-12-04T12:26:02.6908835Z file:/home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-12-04T12:26:02.6924952Z Entering 'third_party/FP16'
2025-12-04T12:26:02.6936306Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-12-04T12:26:02.6948323Z Entering 'third_party/FXdiv'
2025-12-04T12:26:02.6962224Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-12-04T12:26:02.6978608Z Entering 'third_party/NNPACK'
2025-12-04T12:26:02.6990963Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-12-04T12:26:02.7001369Z Entering 'third_party/NVTX'
2025-12-04T12:26:02.7015662Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-12-04T12:26:02.7026244Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:02.7042564Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-12-04T12:26:02.7052136Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:02.7064478Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-12-04T12:26:02.7080329Z Entering 'third_party/aiter'
2025-12-04T12:26:02.7093350Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-12-04T12:26:02.7104157Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:02.7119211Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-12-04T12:26:02.7140128Z Entering 'third_party/benchmark'
2025-12-04T12:26:02.7151858Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:02.7164096Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:02.7180160Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-12-04T12:26:02.7193162Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:02.7209800Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-12-04T12:26:02.7221356Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:02.7232967Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-12-04T12:26:02.7247762Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:02.7258584Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-12-04T12:26:02.7268778Z Entering 'third_party/cutlass'
2025-12-04T12:26:02.7280175Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-12-04T12:26:02.7297742Z Entering 'third_party/fbgemm'
2025-12-04T12:26:02.7308953Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-12-04T12:26:02.7318185Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:02.7331820Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-12-04T12:26:02.7343338Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:02.7355499Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-12-04T12:26:02.7370258Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:02.7383164Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-12-04T12:26:02.7392443Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:02.7403070Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-12-04T12:26:02.7422454Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:02.7434674Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-12-04T12:26:02.7449535Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:02.7462836Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-12-04T12:26:02.7471872Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:02.7482498Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-12-04T12:26:02.7494628Z Entering 'third_party/flash-attention'
2025-12-04T12:26:02.7506678Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-12-04T12:26:02.7515680Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:02.7527479Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-12-04T12:26:02.7545748Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:02.7564541Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-12-04T12:26:02.7579200Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:02.7592759Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-12-04T12:26:02.7603277Z Entering 'third_party/fmt'
2025-12-04T12:26:02.7614269Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:02.7623967Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:02.7634997Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-12-04T12:26:02.7644565Z Entering 'third_party/gloo'
2025-12-04T12:26:02.7654917Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-12-04T12:26:02.7671007Z Entering 'third_party/googletest'
2025-12-04T12:26:02.7682918Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:02.7695333Z Entering 'third_party/ideep'
2025-12-04T12:26:02.7710175Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-12-04T12:26:02.7719952Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:02.7729893Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-12-04T12:26:02.7742558Z Entering 'third_party/ittapi'
2025-12-04T12:26:02.7757512Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-12-04T12:26:02.7767567Z Entering 'third_party/kineto'
2025-12-04T12:26:02.7779215Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-12-04T12:26:02.7795010Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:02.7806525Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-12-04T12:26:02.7818852Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:02.7831571Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-12-04T12:26:02.7846375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:02.7859790Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-12-04T12:26:02.7870246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:02.7882571Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:02.7894168Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:02.7905544Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-12-04T12:26:02.7918176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:02.7929618Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-12-04T12:26:02.7941017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:02.7957214Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-12-04T12:26:02.7966074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:02.7977315Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:02.7987011Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:02.7996889Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-12-04T12:26:02.8006202Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:02.8021908Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-12-04T12:26:02.8032539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:02.8044193Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T12:26:02.8060622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:02.8075928Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T12:26:02.8085816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:02.8099288Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T12:26:02.8112402Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:02.8129410Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:02.8138923Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:02.8156726Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:02.8168598Z Entering 'third_party/kleidiai'
2025-12-04T12:26:02.8179080Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-12-04T12:26:02.8188841Z Entering 'third_party/mimalloc'
2025-12-04T12:26:02.8206356Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-12-04T12:26:02.8216390Z Entering 'third_party/nlohmann'
2025-12-04T12:26:02.8230901Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-12-04T12:26:02.8245893Z Entering 'third_party/onnx'
2025-12-04T12:26:02.8257780Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-12-04T12:26:02.8275693Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:02.8287353Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:02.8303726Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:02.8317078Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-12-04T12:26:02.8329090Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:02.8339416Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:02.8351596Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:02.8363068Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:02.8372326Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:02.8384413Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-12-04T12:26:02.8392805Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:02.8407314Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-12-04T12:26:02.8416564Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:02.8432386Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-12-04T12:26:02.8441634Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:02.8451936Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-12-04T12:26:02.8460992Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:02.8472546Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T12:26:02.8481455Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:02.8492417Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T12:26:02.8503614Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:02.8514150Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T12:26:02.8528396Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:02.8539972Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-12-04T12:26:02.8558765Z Entering 'third_party/pocketfft'
2025-12-04T12:26:02.8573213Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-12-04T12:26:02.8582828Z Entering 'third_party/protobuf'
2025-12-04T12:26:02.8595811Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-12-04T12:26:02.8607601Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:02.8625671Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:02.8639437Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:02.8651563Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:02.8663365Z Entering 'third_party/psimd'
2025-12-04T12:26:02.8676088Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-12-04T12:26:02.8685172Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:02.8703169Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-12-04T12:26:02.8712953Z Entering 'third_party/pybind11'
2025-12-04T12:26:02.8728224Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:02.8737499Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:02.8752535Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-12-04T12:26:02.8761895Z Entering 'third_party/sleef'
2025-12-04T12:26:02.8772481Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-12-04T12:26:02.8783827Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:02.8794204Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-12-04T12:26:02.8803154Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:02.8814508Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:02.8823854Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:02.8833607Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-12-04T12:26:02.8841995Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:02.8850880Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-12-04T12:26:02.8860148Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:02.8871764Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:02.8881157Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:02.8891826Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-12-04T12:26:02.8922372Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.8943078Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.8961626Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.8978957Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9018543Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9019248Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9022864Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9037192Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9050694Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9064326Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9079950Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9092897Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9109776Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9122316Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9137340Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9151241Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9164177Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9177730Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9191631Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9204891Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9223164Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9238613Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9251868Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9264970Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9278970Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9296486Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9310454Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9326082Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9340516Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9357221Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9376184Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9392247Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9406362Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9425726Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9445534Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9467865Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9485794Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9500634Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9514757Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9532246Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9546856Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9559821Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9572848Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9586692Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9601338Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9620083Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9635998Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9654326Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9668389Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9683087Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9697713Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9711571Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9726029Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9741431Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9756171Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9770962Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9787460Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9802486Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9817736Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9832580Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9852162Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9866162Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9882840Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9898547Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9913515Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9928506Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9944251Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9961604Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9977552Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:02.9991826Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0012339Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0026489Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0043457Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0060695Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0077206Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0093176Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0107819Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0123403Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0137280Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0151669Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0169229Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:03.0195598Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-12-04T12:26:03.0219309Z ##[endgroup]
2025-12-04T12:26:03.0219489Z ##[group]Fetching the repository
2025-12-04T12:26:03.0223032Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-12-04T12:26:06.7513225Z From https://github.com/pytorch/pytorch
2025-12-04T12:26:06.7513508Z  * [new branch]                2.6.0.dev20241004+      -> origin/2.6.0.dev20241004+
2025-12-04T12:26:06.7513740Z  * [new branch]                2.9.1                   -> origin/2.9.1
2025-12-04T12:26:06.7513957Z  * [new branch]                AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest
2025-12-04T12:26:06.7514192Z  * [new branch]                Flamefire-patch-1       -> origin/Flamefire-patch-1
2025-12-04T12:26:06.7514406Z  * [new branch]                HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes
2025-12-04T12:26:06.7514681Z  * [new branch]                HOPrintFunc             -> origin/HOPrintFunc
2025-12-04T12:26:06.7514866Z  * [new branch]                IvanKobzarev/stack/1    -> origin/IvanKobzarev/stack/1
2025-12-04T12:26:06.7515059Z  * [new branch]                NicoshevSVE128          -> origin/NicoshevSVE128
2025-12-04T12:26:06.7515249Z  * [new branch]                PR-AOTInductorNoneBug   -> origin/PR-AOTInductorNoneBug
2025-12-04T12:26:06.7515460Z  * [new branch]                PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix
2025-12-04T12:26:06.7515671Z  * [new branch]                PR-FixConfigsIssue      -> origin/PR-FixConfigsIssue
2025-12-04T12:26:06.7515862Z  * [new branch]                PR-NoneBugFix-viable    -> origin/PR-NoneBugFix-viable
2025-12-04T12:26:06.7516046Z  * [new branch]                PR-ResetToZero          -> origin/PR-ResetToZero
2025-12-04T12:26:06.7516243Z  * [new branch]                Update-Flash-Packaging  -> origin/Update-Flash-Packaging
2025-12-04T12:26:06.7516429Z  * [new branch]                VLA_exp                 -> origin/VLA_exp
2025-12-04T12:26:06.7516598Z  * [new branch]                activation_bench        -> origin/activation_bench
2025-12-04T12:26:06.7516836Z  * [new branch]                addmm-heuristic         -> origin/addmm-heuristic
2025-12-04T12:26:06.7517015Z  * [new branch]                adi/onednn_aarch64      -> origin/adi/onednn_aarch64
2025-12-04T12:26:06.7517187Z  * [new branch]                adi/test                -> origin/adi/test
2025-12-04T12:26:06.7517355Z  * [new branch]                adi/test_bgemm          -> origin/adi/test_bgemm
2025-12-04T12:26:06.7517525Z  * [new branch]                adi/test_m8g            -> origin/adi/test_m8g
2025-12-04T12:26:06.7517698Z  * [new branch]                adi/test_onednn         -> origin/adi/test_onednn
2025-12-04T12:26:06.7517872Z  * [new branch]                adi/test_onednn_v3.9    -> origin/adi/test_onednn_v3.9
2025-12-04T12:26:06.7518070Z  * [new branch]                adi/test_presve_change  -> origin/adi/test_presve_change
2025-12-04T12:26:06.7518253Z  * [new branch]                adi/test_timm           -> origin/adi/test_timm
2025-12-04T12:26:06.7518439Z  * [new branch]                adi/testpresve_change   -> origin/adi/testpresve_change
2025-12-04T12:26:06.7519120Z  * [new branch]                aditew01/test/vec_bf16  -> origin/aditew01/test/vec_bf16
2025-12-04T12:26:06.7519324Z  * [new branch]                ah-globalfeedback-hook  -> origin/ah-globalfeedback-hook
2025-12-04T12:26:06.7519691Z  * [new branch]                albanD-patch-1          -> origin/albanD-patch-1
2025-12-04T12:26:06.7519884Z  * [new branch]                also-surround-shimh     -> origin/also-surround-shimh
2025-12-04T12:26:06.7520075Z  * [new branch]                angelayi/aot_compile    -> origin/angelayi/aot_compile
2025-12-04T12:26:06.7520295Z  * [new branch]                angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files
2025-12-04T12:26:06.7520512Z  * [new branch]                angelayi/benchmark      -> origin/angelayi/benchmark
2025-12-04T12:26:06.7520739Z  * [new branch]                angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization
2025-12-04T12:26:06.7520976Z  * [new branch]                angelayi/cpp_loader     -> origin/angelayi/cpp_loader
2025-12-04T12:26:06.7521168Z  * [new branch]                angelayi/inductor_const -> origin/angelayi/inductor_const
2025-12-04T12:26:06.7521357Z  * [new branch]                angelayi/lstm           -> origin/angelayi/lstm
2025-12-04T12:26:06.7521546Z  * [new branch]                angelayi/no_so_weight   -> origin/angelayi/no_so_weight
2025-12-04T12:26:06.7521734Z  * [new branch]                angelayi/scan_layers    -> origin/angelayi/scan_layers
2025-12-04T12:26:06.7521925Z  * [new branch]                angelayi/side_eff       -> origin/angelayi/side_eff
2025-12-04T12:26:06.7522105Z  * [new branch]                angelayi/state_dict     -> origin/angelayi/state_dict
2025-12-04T12:26:06.7522291Z  * [new branch]                angelayi/symint_input   -> origin/angelayi/symint_input
2025-12-04T12:26:06.7522482Z  * [new branch]                angelayi/symm_mem       -> origin/angelayi/symm_mem
2025-12-04T12:26:06.7522668Z  * [new branch]                angelayi/test_cpp       -> origin/angelayi/test_cpp
2025-12-04T12:26:06.7522849Z  * [new branch]                angelayi/torch_size     -> origin/angelayi/torch_size
2025-12-04T12:26:06.7523034Z  * [new branch]                annotate_assert         -> origin/annotate_assert
2025-12-04T12:26:06.7523232Z  * [new branch]                annotate_fallback_kernel -> origin/annotate_fallback_kernel
2025-12-04T12:26:06.7523434Z  * [new branch]                annotation_deepcopy     -> origin/annotation_deepcopy
2025-12-04T12:26:06.7523617Z  * [new branch]                annotation_dynamo       -> origin/annotation_dynamo
2025-12-04T12:26:06.7523799Z  * [new branch]                aot_eager_stack_trace   -> origin/aot_eager_stack_trace
2025-12-04T12:26:06.7523980Z  * [new branch]                aoti-cuda-alloc         -> origin/aoti-cuda-alloc
2025-12-04T12:26:06.7524162Z  * [new branch]                aoti_const_device       -> origin/aoti_const_device
2025-12-04T12:26:06.7524354Z  * [new branch]                aoti_fqn_name_interface -> origin/aoti_fqn_name_interface
2025-12-04T12:26:06.7524566Z  * [new branch]                aoti_package_weights_binary -> origin/aoti_package_weights_binary
2025-12-04T12:26:06.7524769Z  * [new branch]                aoti_target_windows     -> origin/aoti_target_windows
2025-12-04T12:26:06.7525001Z  * [new branch]                arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling
2025-12-04T12:26:06.7525211Z  * [new branch]                async_tp                -> origin/async_tp
2025-12-04T12:26:06.7525416Z  * [new branch]                atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124
2025-12-04T12:26:06.7525659Z  * [new branch]                atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1
2025-12-04T12:26:06.7525879Z  * [new branch]                atalman-patch-2         -> origin/atalman-patch-2
2025-12-04T12:26:06.7526059Z  * [new branch]                atalman-patch-3         -> origin/atalman-patch-3
2025-12-04T12:26:06.7526261Z  * [new branch]                atalman-patch-4         -> origin/atalman-patch-4
2025-12-04T12:26:06.7526438Z  * [new branch]                atalman-patch-5         -> origin/atalman-patch-5
2025-12-04T12:26:06.7526703Z  * [new branch]                atalman-patch-6         -> origin/atalman-patch-6
2025-12-04T12:26:06.7526938Z  * [new branch]                atalman-patch-7         -> origin/atalman-patch-7
2025-12-04T12:26:06.7527117Z  * [new branch]                atalman-patch-8         -> origin/atalman-patch-8
2025-12-04T12:26:06.7527301Z  * [new branch]                atalman_inductor_2.3.1  -> origin/atalman_inductor_2.3.1
2025-12-04T12:26:06.7527494Z  * [new branch]                atalman_inductor_2.4.0  -> origin/atalman_inductor_2.4.0
2025-12-04T12:26:06.7527685Z  * [new branch]                atalman_inductor_2.4.x  -> origin/atalman_inductor_2.4.x
2025-12-04T12:26:06.7527896Z  * [new branch]                attention_benchmarking_clean -> origin/attention_benchmarking_clean
2025-12-04T12:26:06.7528116Z  * [new branch]                bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add
2025-12-04T12:26:06.7528315Z  * [new branch]                bahuang/fix_debug_mode  -> origin/bahuang/fix_debug_mode
2025-12-04T12:26:06.7528511Z  * [new branch]                bahuang/fix_expand      -> origin/bahuang/fix_expand
2025-12-04T12:26:06.7528687Z  * [new branch]                bahuang/test            -> origin/bahuang/test
2025-12-04T12:26:06.7528851Z  * [new branch]                base/1.5                -> origin/base/1.5
2025-12-04T12:26:06.7529055Z  * [new branch]                batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention
2025-12-04T12:26:06.7529271Z  * [new branch]                bench_scaled_mm_ops     -> origin/bench_scaled_mm_ops
2025-12-04T12:26:06.7529457Z  * [new branch]                benchmark-updates       -> origin/benchmark-updates
2025-12-04T12:26:06.7529646Z  * [new branch]                benchmarking-script     -> origin/benchmarking-script
2025-12-04T12:26:06.7529846Z  * [new branch]                bertmaher/pinbump26     -> origin/bertmaher/pinbump26
2025-12-04T12:26:06.7530226Z  * [new branch]                bertrand/cutlass        -> origin/bertrand/cutlass
2025-12-04T12:26:06.7530416Z  * [new branch]                bf/bug-static-input     -> origin/bf/bug-static-input
2025-12-04T12:26:06.7530597Z  * [new branch]                bf/cg-backend           -> origin/bf/cg-backend
2025-12-04T12:26:06.7530775Z  * [new branch]                bf/cg-nccl-test         -> origin/bf/cg-nccl-test
2025-12-04T12:26:06.7530959Z  * [new branch]                bf/cg-remove-check      -> origin/bf/cg-remove-check
2025-12-04T12:26:06.7531155Z  * [new branch]                bf/clean-torchbench-hf  -> origin/bf/clean-torchbench-hf
2025-12-04T12:26:06.7531348Z  * [new branch]                bf/combo-debug-log      -> origin/bf/combo-debug-log
2025-12-04T12:26:06.7531521Z  * [new branch]                bf/cudagraph            -> origin/bf/cudagraph
2025-12-04T12:26:06.7531752Z  * [new branch]                bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation
2025-12-04T12:26:06.7532112Z  * [new branch]                bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark
2025-12-04T12:26:06.7532422Z  * [new branch]                bf/cudagraph-partition  -> origin/bf/cudagraph-partition
2025-12-04T12:26:06.7532625Z  * [new branch]                bf/donated-buffer-bench -> origin/bf/donated-buffer-bench
2025-12-04T12:26:06.7532824Z  * [new branch]                bf/dynamo-partition     -> origin/bf/dynamo-partition
2025-12-04T12:26:06.7533002Z  * [new branch]                bf/lite                 -> origin/bf/lite
2025-12-04T12:26:06.7533179Z  * [new branch]                bf/pa-non-divisible     -> origin/bf/pa-non-divisible
2025-12-04T12:26:06.7533406Z  * [new branch]                bf/partition-cache-free-symbols -> origin/bf/partition-cache-free-symbols
2025-12-04T12:26:06.7534148Z  * [new branch]                bf/partition-memory-plan -> origin/bf/partition-memory-plan
2025-12-04T12:26:06.7534359Z  * [new branch]                bf/partition-move-cpu   -> origin/bf/partition-move-cpu
2025-12-04T12:26:06.7534615Z  * [new branch]                bf/partition-view-fallback -> origin/bf/partition-view-fallback
2025-12-04T12:26:06.7534835Z  * [new branch]                bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d
2025-12-04T12:26:06.7535032Z  * [new branch]                bf/timm-nov-26-2025     -> origin/bf/timm-nov-26-2025
2025-12-04T12:26:06.7535243Z  * [new branch]                bf/transformer-pin-4-57-3 -> origin/bf/transformer-pin-4-57-3
2025-12-04T12:26:06.7535466Z  * [new branch]                bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492
2025-12-04T12:26:06.7535691Z  * [new branch]                bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb
2025-12-04T12:26:06.7535908Z  * [new branch]                bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129
2025-12-04T12:26:06.7536116Z  * [new branch]                bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d
2025-12-04T12:26:06.7536331Z  * [new branch]                bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e
2025-12-04T12:26:06.7536545Z  * [new branch]                bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c
2025-12-04T12:26:06.7536814Z  * [new branch]                bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c
2025-12-04T12:26:06.7537035Z  * [new branch]                bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f
2025-12-04T12:26:06.7537253Z  * [new branch]                bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0
2025-12-04T12:26:06.7537466Z  * [new branch]                bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149
2025-12-04T12:26:06.7537683Z  * [new branch]                bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a
2025-12-04T12:26:06.7537898Z  * [new branch]                bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b
2025-12-04T12:26:06.7538111Z  * [new branch]                bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new
2025-12-04T12:26:06.7538326Z  * [new branch]                bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8
2025-12-04T12:26:06.7538536Z  * [new branch]                bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2
2025-12-04T12:26:06.7538743Z  * [new branch]                bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563
2025-12-04T12:26:06.7538950Z  * [new branch]                brister/fx_device_type  -> origin/brister/fx_device_type
2025-12-04T12:26:06.7539165Z  * [new branch]                brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx
2025-12-04T12:26:06.7539413Z  * [new branch]                brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check
2025-12-04T12:26:06.7539639Z  * [new branch]                bwd-backup              -> origin/bwd-backup
2025-12-04T12:26:06.7539805Z  * [new branch]                c57382a49               -> origin/c57382a49
2025-12-04T12:26:06.7539969Z  * [new branch]                ca_0431d47eaa           -> origin/ca_0431d47eaa
2025-12-04T12:26:06.7540142Z  * [new branch]                ca_fix_0431d47eaa       -> origin/ca_fix_0431d47eaa
2025-12-04T12:26:06.7540341Z  * [new branch]                camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push
2025-12-04T12:26:06.7540543Z  * [new branch]                cccclai-patch-1         -> origin/cccclai-patch-1
2025-12-04T12:26:06.7540779Z  * [new branch]                cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_
2025-12-04T12:26:06.7541095Z  * [new branch]                cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_
2025-12-04T12:26:06.7541376Z  * [new branch]                cherry-pick-162208-by-pytorch_bot_bot_ -> origin/cherry-pick-162208-by-pytorch_bot_bot_
2025-12-04T12:26:06.7541649Z  * [new branch]                cherry-pick-163169-by-pytorch_bot_bot_ -> origin/cherry-pick-163169-by-pytorch_bot_bot_
2025-12-04T12:26:06.7541965Z  * [new branch]                cherry-pick-165086-by-pytorch_bot_bot_ -> origin/cherry-pick-165086-by-pytorch_bot_bot_
2025-12-04T12:26:06.7542242Z  * [new branch]                cherry-pick-165514-by-pytorch_bot_bot_ -> origin/cherry-pick-165514-by-pytorch_bot_bot_
2025-12-04T12:26:06.7542514Z  * [new branch]                cherry-pick-165601-by-pytorch_bot_bot_ -> origin/cherry-pick-165601-by-pytorch_bot_bot_
2025-12-04T12:26:06.7542786Z  * [new branch]                cherry-pick-165667-by-pytorch_bot_bot_ -> origin/cherry-pick-165667-by-pytorch_bot_bot_
2025-12-04T12:26:06.7543062Z  * [new branch]                cherry-pick-165815-by-pytorch_bot_bot_ -> origin/cherry-pick-165815-by-pytorch_bot_bot_
2025-12-04T12:26:06.7543334Z  * [new branch]                cherry-pick-165922-by-pytorch_bot_bot_ -> origin/cherry-pick-165922-by-pytorch_bot_bot_
2025-12-04T12:26:06.7543604Z  * [new branch]                cherry-pick-166148-by-pytorch_bot_bot_ -> origin/cherry-pick-166148-by-pytorch_bot_bot_
2025-12-04T12:26:06.7543875Z  * [new branch]                cherry-pick-166181-by-pytorch_bot_bot_ -> origin/cherry-pick-166181-by-pytorch_bot_bot_
2025-12-04T12:26:06.7544147Z  * [new branch]                cherry-pick-166404-by-pytorch_bot_bot_ -> origin/cherry-pick-166404-by-pytorch_bot_bot_
2025-12-04T12:26:06.7544426Z  * [new branch]                cherry-pick-166427-by-pytorch_bot_bot_ -> origin/cherry-pick-166427-by-pytorch_bot_bot_
2025-12-04T12:26:06.7544695Z  * [new branch]                cherry-pick-166480-by-pytorch_bot_bot_ -> origin/cherry-pick-166480-by-pytorch_bot_bot_
2025-12-04T12:26:06.7544969Z  * [new branch]                cherry-pick-166570-by-pytorch_bot_bot_ -> origin/cherry-pick-166570-by-pytorch_bot_bot_
2025-12-04T12:26:06.7545244Z  * [new branch]                cherry-pick-166993-by-pytorch_bot_bot_ -> origin/cherry-pick-166993-by-pytorch_bot_bot_
2025-12-04T12:26:06.7545516Z  * [new branch]                cherry-pick-167111-by-pytorch_bot_bot_ -> origin/cherry-pick-167111-by-pytorch_bot_bot_
2025-12-04T12:26:06.7545786Z  * [new branch]                cherry-pick-167478-by-pytorch_bot_bot_ -> origin/cherry-pick-167478-by-pytorch_bot_bot_
2025-12-04T12:26:06.7546028Z  * [new branch]                cherry_pick_166036_166040 -> origin/cherry_pick_166036_166040
2025-12-04T12:26:06.7546217Z  * [new branch]                cherry_pick_166457      -> origin/cherry_pick_166457
2025-12-04T12:26:06.7546398Z  * [new branch]                cherrypick_166338       -> origin/cherrypick_166338
2025-12-04T12:26:06.7546575Z  * [new branch]                cherrypick_166458       -> origin/cherrypick_166458
2025-12-04T12:26:06.7546814Z  * [new branch]                cherrypick_166586       -> origin/cherrypick_166586
2025-12-04T12:26:06.7547001Z  * [new branch]                cherrypick_166956       -> origin/cherrypick_166956
2025-12-04T12:26:06.7547174Z  * [new branch]                ci_attn                 -> origin/ci_attn
2025-12-04T12:26:06.7547352Z  * [new branch]                codex-testing           -> origin/codex-testing
2025-12-04T12:26:06.7547622Z  * [new branch]                codex/add-check_memory_overlap-helper-functions -> origin/codex/add-check_memory_overlap-helper-functions
2025-12-04T12:26:06.7547927Z  * [new branch]                codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch
2025-12-04T12:26:06.7548248Z  * [new branch]                codex/investigate-segfaults-in-get_tensor_storage_id -> origin/codex/investigate-segfaults-in-get_tensor_storage_id
2025-12-04T12:26:06.7548624Z  * [new branch]                codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run
2025-12-04T12:26:06.7548933Z  * [new branch]                compatiblpy39util       -> origin/compatiblpy39util
2025-12-04T12:26:06.7549115Z  * [new branch]                cond_hop_device         -> origin/cond_hop_device
2025-12-04T12:26:06.7549325Z  * [new branch]                context_test            -> origin/context_test
2025-12-04T12:26:06.7549566Z  * [new branch]                copilot/code-style-cleanup-python-pip -> origin/copilot/code-style-cleanup-python-pip
2025-12-04T12:26:06.7549817Z  * [new branch]                cpio/fix_new_ami_tests  -> origin/cpio/fix_new_ami_tests
2025-12-04T12:26:06.7550047Z  * [new branch]                cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade
2025-12-04T12:26:06.7550306Z  * [new branch]                crpa/typo-in-inductor_comm_lowering -> origin/crpa/typo-in-inductor_comm_lowering
2025-12-04T12:26:06.7550541Z  * [new branch]                csl/always_produce_xml  -> origin/csl/always_produce_xml
2025-12-04T12:26:06.7550754Z  * [new branch]                csl/build_test_more_procs -> origin/csl/build_test_more_procs
2025-12-04T12:26:06.7550969Z  * [new branch]                csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2
2025-12-04T12:26:06.7551160Z  * [new branch]                csl/clean_up            -> origin/csl/clean_up
2025-12-04T12:26:06.7551360Z  * [new branch]                csl/fix_retry_segfault_exit -> origin/csl/fix_retry_segfault_exit
2025-12-04T12:26:06.7551559Z  * [new branch]                csl/katex               -> origin/csl/katex
2025-12-04T12:26:06.7551730Z  * [new branch]                csl/larger_runner       -> origin/csl/larger_runner
2025-12-04T12:26:06.7551916Z  * [new branch]                csl/lint_testing        -> origin/csl/lint_testing
2025-12-04T12:26:06.7552095Z  * [new branch]                csl/lint_thing          -> origin/csl/lint_thing
2025-12-04T12:26:06.7552279Z  * [new branch]                csl/lintrunner_stuff    -> origin/csl/lintrunner_stuff
2025-12-04T12:26:06.7552483Z  * [new branch]                csl/manually_gen_json   -> origin/csl/manually_gen_json
2025-12-04T12:26:06.7552671Z  * [new branch]                csl/mps_sharding        -> origin/csl/mps_sharding
2025-12-04T12:26:06.7552856Z  * [new branch]                csl/multistage_docker   -> origin/csl/multistage_docker
2025-12-04T12:26:06.7553049Z  * [new branch]                csl/print_timing        -> origin/csl/print_timing
2025-12-04T12:26:06.7553233Z  * [new branch]                csl/remove_experiment   -> origin/csl/remove_experiment
2025-12-04T12:26:06.7553444Z  * [new branch]                csl/remove_maybe_unused_var -> origin/csl/remove_maybe_unused_var
2025-12-04T12:26:06.7553682Z  * [new branch]                csl/remove_repo_specific_autolabel -> origin/csl/remove_repo_specific_autolabel
2025-12-04T12:26:06.7553907Z  * [new branch]                csl/remove_run_parallel -> origin/csl/remove_run_parallel
2025-12-04T12:26:06.7554105Z  * [new branch]                csl/remove_unused_vars  -> origin/csl/remove_unused_vars
2025-12-04T12:26:06.7554297Z  * [new branch]                csl/revert_open         -> origin/csl/revert_open
2025-12-04T12:26:06.7554472Z  * [new branch]                csl/skip_build          -> origin/csl/skip_build
2025-12-04T12:26:06.7554673Z  * [new branch]                csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs
2025-12-04T12:26:06.7554872Z  * [new branch]                csl/td_job_level        -> origin/csl/td_job_level
2025-12-04T12:26:06.7555081Z  * [new branch]                csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner
2025-12-04T12:26:06.7555334Z  * [new branch]                csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn
2025-12-04T12:26:06.7555590Z  * [new branch]                csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence
2025-12-04T12:26:06.7555811Z  * [new branch]                csl/upload_json_running -> origin/csl/upload_json_running
2025-12-04T12:26:06.7556032Z  * [new branch]                csl/win_sccache         -> origin/csl/win_sccache
2025-12-04T12:26:06.7556206Z  * [new branch]                csl/xml_stuff           -> origin/csl/xml_stuff
2025-12-04T12:26:06.7556377Z  * [new branch]                cublasrelax2            -> origin/cublasrelax2
2025-12-04T12:26:06.7556580Z  * [new branch]                cuda_mempool            -> origin/cuda_mempool
2025-12-04T12:26:06.7556819Z  * [new branch]                custom_lowering_dict    -> origin/custom_lowering_dict
2025-12-04T12:26:06.7557027Z  * [new branch]                d4l3k/debug_plane_frtrace -> origin/d4l3k/debug_plane_frtrace
2025-12-04T12:26:06.7557221Z  * [new branch]                daxia6/2.8o3            -> origin/daxia6/2.8o3
2025-12-04T12:26:06.7557394Z  * [new branch]                debug-guard             -> origin/debug-guard
2025-12-04T12:26:06.7557574Z  * [new branch]                delete-quant-docs       -> origin/delete-quant-docs
2025-12-04T12:26:06.7557907Z  * [new branch]                dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0
2025-12-04T12:26:06.7558366Z  * [new branch]                dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1
2025-12-04T12:26:06.7558704Z  * [new branch]                desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper
2025-12-04T12:26:06.7558956Z  * [new branch]                desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64
2025-12-04T12:26:06.7559191Z  * [new branch]                dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt
2025-12-04T12:26:06.7559394Z  * [new branch]                dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd
2025-12-04T12:26:06.7559591Z  * [new branch]                dev/joona/Unranked      -> origin/dev/joona/Unranked
2025-12-04T12:26:06.7559778Z  * [new branch]                dev/joona/cat           -> origin/dev/joona/cat
2025-12-04T12:26:06.7559962Z  * [new branch]                dev/joona/embeddingbag  -> origin/dev/joona/embeddingbag
2025-12-04T12:26:06.7560173Z  * [new branch]                dev/joona/fix_sdpa_memtest -> origin/dev/joona/fix_sdpa_memtest
2025-12-04T12:26:06.7560398Z  * [new branch]                dev/joona/getTensorsString -> origin/dev/joona/getTensorsString
2025-12-04T12:26:06.7560621Z  * [new branch]                dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14
2025-12-04T12:26:06.7560842Z  * [new branch]                dev/joona/scalar_clamp  -> origin/dev/joona/scalar_clamp
2025-12-04T12:26:06.7561031Z  * [new branch]                dev/joona/sdpa          -> origin/dev/joona/sdpa
2025-12-04T12:26:06.7561211Z  * [new branch]                dev/joona/sdpa_api      -> origin/dev/joona/sdpa_api
2025-12-04T12:26:06.7561399Z  * [new branch]                dev/joona/type_inf      -> origin/dev/joona/type_inf
2025-12-04T12:26:06.7561604Z  * [new branch]                dev/joona/ulpAssertClose -> origin/dev/joona/ulpAssertClose
2025-12-04T12:26:06.7561799Z  * [new branch]                dev/joona/upsize3d      -> origin/dev/joona/upsize3d
2025-12-04T12:26:06.7561986Z  * [new branch]                disp_counter            -> origin/disp_counter
2025-12-04T12:26:06.7562164Z  * [new branch]                divyanshk-patch-1       -> origin/divyanshk-patch-1
2025-12-04T12:26:06.7562345Z  * [new branch]                docs                    -> origin/docs
2025-12-04T12:26:06.7562522Z  * [new branch]                documentation           -> origin/documentation
2025-12-04T12:26:06.7562705Z  * [new branch]                eager_model_benchmarks  -> origin/eager_model_benchmarks
2025-12-04T12:26:06.7562920Z  * [new branch]                embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control
2025-12-04T12:26:06.7563154Z  * [new branch]                embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B
2025-12-04T12:26:06.7563407Z  * [new branch]                embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B
2025-12-04T12:26:06.7563613Z  * [new branch]                eqy-patch-1             -> origin/eqy-patch-1
2025-12-04T12:26:06.7563820Z  * [new branch]                eqy-patch-2             -> origin/eqy-patch-2
2025-12-04T12:26:06.7563986Z  * [new branch]                eqy-patch-3             -> origin/eqy-patch-3
2025-12-04T12:26:06.7564158Z  * [new branch]                eqy-patch-4             -> origin/eqy-patch-4
2025-12-04T12:26:06.7564333Z  * [new branch]                eqy-patch-5             -> origin/eqy-patch-5
2025-12-04T12:26:06.7564496Z  * [new branch]                eqy-patch-6             -> origin/eqy-patch-6
2025-12-04T12:26:06.7564679Z  * [new branch]                exclamaforte/amd-ma     -> origin/exclamaforte/amd-ma
2025-12-04T12:26:06.7564918Z  * [new branch]                exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run
2025-12-04T12:26:06.7565176Z  * [new branch]                exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor
2025-12-04T12:26:06.7565438Z  * [new branch]                exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion
2025-12-04T12:26:06.7565727Z  * [new branch]                exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning
2025-12-04T12:26:06.7566021Z  * [new branch]                exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg
2025-12-04T12:26:06.7566334Z  * [new branch]                exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run
2025-12-04T12:26:06.7566605Z  * [new branch]                exclamaforte/fusion-data -> origin/exclamaforte/fusion-data
2025-12-04T12:26:06.7566901Z  * [new branch]                exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run
2025-12-04T12:26:06.7567154Z  * [new branch]                exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model
2025-12-04T12:26:06.7567390Z  * [new branch]                exclamaforte/gemm-model -> origin/exclamaforte/gemm-model
2025-12-04T12:26:06.7567659Z  * [new branch]                exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection
2025-12-04T12:26:06.7567935Z  * [new branch]                exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd
2025-12-04T12:26:06.7568161Z  * [new branch]                exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model
2025-12-04T12:26:06.7568433Z  * [new branch]                exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor
2025-12-04T12:26:06.7568711Z  * [new branch]                exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo
2025-12-04T12:26:06.7568977Z  * [new branch]                exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization
2025-12-04T12:26:06.7569250Z  * [new branch]                exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode
2025-12-04T12:26:06.7569526Z  * [new branch]                exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs
2025-12-04T12:26:06.7569823Z  * [new branch]                exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2
2025-12-04T12:26:06.7570050Z  * [new branch]                exec                    -> origin/exec
2025-12-04T12:26:06.7570239Z  * [new branch]                experimental-mosaic     -> origin/experimental-mosaic
2025-12-04T12:26:06.7570434Z  * [new branch]                export-D61047529        -> origin/export-D61047529
2025-12-04T12:26:06.7570699Z  * [new branch]                export-D71412006        -> origin/export-D71412006
2025-12-04T12:26:06.7570882Z  * [new branch]                export-D73042989        -> origin/export-D73042989
2025-12-04T12:26:06.7571114Z  * [new branch]                export-D78957093        -> origin/export-D78957093
2025-12-04T12:26:06.7571289Z  * [new branch]                export-D78996107        -> origin/export-D78996107
2025-12-04T12:26:06.7571465Z  * [new branch]                export-D80823877        -> origin/export-D80823877
2025-12-04T12:26:06.7571672Z  * [new branch]                export-D80958642        -> origin/export-D80958642
2025-12-04T12:26:06.7571845Z  * [new branch]                export-D81054193        -> origin/export-D81054193
2025-12-04T12:26:06.7572027Z  * [new branch]                export-D81204584        -> origin/export-D81204584
2025-12-04T12:26:06.7572201Z  * [new branch]                export-D81429090        -> origin/export-D81429090
2025-12-04T12:26:06.7572378Z  * [new branch]                export-D82250826        -> origin/export-D82250826
2025-12-04T12:26:06.7572556Z  * [new branch]                export-D82253817        -> origin/export-D82253817
2025-12-04T12:26:06.7572729Z  * [new branch]                export-D83541846        -> origin/export-D83541846
2025-12-04T12:26:06.7572909Z  * [new branch]                export-D83627170        -> origin/export-D83627170
2025-12-04T12:26:06.7573085Z  * [new branch]                export-D83766701        -> origin/export-D83766701
2025-12-04T12:26:06.7573258Z  * [new branch]                export-D83768878        -> origin/export-D83768878
2025-12-04T12:26:06.7573433Z  * [new branch]                export-D83769447        -> origin/export-D83769447
2025-12-04T12:26:06.7573610Z  * [new branch]                export-D84089824        -> origin/export-D84089824
2025-12-04T12:26:06.7573784Z  * [new branch]                export-D84213020        -> origin/export-D84213020
2025-12-04T12:26:06.7573963Z  * [new branch]                export-D84373821        -> origin/export-D84373821
2025-12-04T12:26:06.7574138Z  * [new branch]                export-D84612194        -> origin/export-D84612194
2025-12-04T12:26:06.7574308Z  * [new branch]                export-D84890985        -> origin/export-D84890985
2025-12-04T12:26:06.7574485Z  * [new branch]                export-D85122326        -> origin/export-D85122326
2025-12-04T12:26:06.7574664Z  * [new branch]                export-D86256198        -> origin/export-D86256198
2025-12-04T12:26:06.7574837Z  * [new branch]                export-D86460608        -> origin/export-D86460608
2025-12-04T12:26:06.7575015Z  * [new branch]                export-D86474796        -> origin/export-D86474796
2025-12-04T12:26:06.7575191Z  * [new branch]                export-D86712396        -> origin/export-D86712396
2025-12-04T12:26:06.7575363Z  * [new branch]                export-D87022129        -> origin/export-D87022129
2025-12-04T12:26:06.7575541Z  * [new branch]                export-D87838959        -> origin/export-D87838959
2025-12-04T12:26:06.7575719Z  * [new branch]                export-D88319437        -> origin/export-D88319437
2025-12-04T12:26:06.7575945Z  * [new branch]                exported-model-train-idempotent -> origin/exported-model-train-idempotent
2025-12-04T12:26:06.7576181Z  * [new branch]                ezyang-titan-october    -> origin/ezyang-titan-october
2025-12-04T12:26:06.7576381Z  * [new branch]                ezyang-titan-october2   -> origin/ezyang-titan-october2
2025-12-04T12:26:06.7576575Z  * [new branch]                ezyang-war              -> origin/ezyang-war
2025-12-04T12:26:06.7576825Z  * [new branch]                ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors
2025-12-04T12:26:06.7577021Z  * [new branch]                fa_u8_brgemm            -> origin/fa_u8_brgemm
2025-12-04T12:26:06.7577210Z  * [new branch]                fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm
2025-12-04T12:26:06.7577409Z  * [new branch]                fastmath_baseline       -> origin/fastmath_baseline
2025-12-04T12:26:06.7577583Z  * [new branch]                fbcode/warm             -> origin/fbcode/warm
2025-12-04T12:26:06.7577746Z  * [new branch]                fca                     -> origin/fca
2025-12-04T12:26:06.7577942Z  * [new branch]                fca2_ca5984c            -> origin/fca2_ca5984c
2025-12-04T12:26:06.7578101Z  * [new branch]                fca5                    -> origin/fca5
2025-12-04T12:26:06.7578282Z  * [new branch]                feature/justknobs-cpp   -> origin/feature/justknobs-cpp
2025-12-04T12:26:06.7578517Z  * [new branch]                feature/numa-forkserver -> origin/feature/numa-forkserver
2025-12-04T12:26:06.7578708Z  * [new branch]                ffast_math_baseline     -> origin/ffast_math_baseline
2025-12-04T12:26:06.7578888Z  * [new branch]                ffast_math_target       -> origin/ffast_math_target
2025-12-04T12:26:06.7579073Z  * [new branch]                findhao/base_commit     -> origin/findhao/base_commit
2025-12-04T12:26:06.7579258Z  * [new branch]                findhao/base_commit1    -> origin/findhao/base_commit1
2025-12-04T12:26:06.7579448Z  * [new branch]                findhao/multistream2    -> origin/findhao/multistream2
2025-12-04T12:26:06.7579636Z  * [new branch]                findhao/multistream5    -> origin/findhao/multistream5
2025-12-04T12:26:06.7579823Z  * [new branch]                findhao/multistream6    -> origin/findhao/multistream6
2025-12-04T12:26:06.7580022Z  * [new branch]                findhao/operatorbench3  -> origin/findhao/operatorbench3
2025-12-04T12:26:06.7580224Z  * [new branch]                findhao/operatorbench5  -> origin/findhao/operatorbench5
2025-12-04T12:26:06.7580417Z  * [new branch]                findhao/tritonparse     -> origin/findhao/tritonparse
2025-12-04T12:26:06.7580630Z  * [new branch]                fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format
2025-12-04T12:26:06.7580842Z  * [new branch]                fix-config-ignore       -> origin/fix-config-ignore
2025-12-04T12:26:06.7581021Z  * [new branch]                fix-dict-guard          -> origin/fix-dict-guard
2025-12-04T12:26:06.7581195Z  * [new branch]                fix_addmm_issue         -> origin/fix_addmm_issue
2025-12-04T12:26:06.7581390Z  * [new branch]                fix_amd_missing_cluster_dims -> origin/fix_amd_missing_cluster_dims
2025-12-04T12:26:06.7581596Z  * [new branch]                fix_bench_bwd_pass      -> origin/fix_bench_bwd_pass
2025-12-04T12:26:06.7581786Z  * [new branch]                fix_mem_profiler_config -> origin/fix_mem_profiler_config
2025-12-04T12:26:06.7581970Z  * [new branch]                fix_nvrtc_discovery     -> origin/fix_nvrtc_discovery
2025-12-04T12:26:06.7582145Z  * [new branch]                fix_op_runner           -> origin/fix_op_runner
2025-12-04T12:26:06.7582313Z  * [new branch]                fix_ubn_159469          -> origin/fix_ubn_159469
2025-12-04T12:26:06.7582486Z  * [new branch]                fixes-triage            -> origin/fixes-triage
2025-12-04T12:26:06.7582658Z  * [new branch]                fixflashinfer           -> origin/fixflashinfer
2025-12-04T12:26:06.7582834Z  * [new branch]                flash_decoding_cpu      -> origin/flash_decoding_cpu
2025-12-04T12:26:06.7583010Z  * [new branch]                flex-flash              -> origin/flex-flash
2025-12-04T12:26:06.7583210Z  * [new branch]                flex_attention_functorch_grad -> origin/flex_attention_functorch_grad
2025-12-04T12:26:06.7583413Z  * [new branch]                flex_flash              -> origin/flex_flash
2025-12-04T12:26:06.7583614Z  * [new branch]                fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule
2025-12-04T12:26:06.7583873Z  * [new branch]                fmassa/tests_comm_compute_scheduler -> origin/fmassa/tests_comm_compute_scheduler
2025-12-04T12:26:06.7584093Z  * [new branch]                forkserver_fix          -> origin/forkserver_fix
2025-12-04T12:26:06.7584277Z  * [new branch]                fsdp2_trace_rules       -> origin/fsdp2_trace_rules
2025-12-04T12:26:06.7584454Z  * [new branch]                fx_cpp                  -> origin/fx_cpp
2025-12-04T12:26:06.7584622Z  * [new branch]                fy/fix-win              -> origin/fy/fix-win
2025-12-04T12:26:06.7584825Z  * [new branch]                galv-patch-1            -> origin/galv-patch-1
2025-12-04T12:26:06.7585060Z  * [new branch]                galv/cudagraphs-conditional-nodes-4 -> origin/galv/cudagraphs-conditional-nodes-4
2025-12-04T12:26:06.7585344Z  * [new branch]                georgehong/cmakelists-patch -> origin/georgehong/cmakelists-patch
2025-12-04T12:26:06.7585558Z  * [new branch]                gh/AlnisM/1/base        -> origin/gh/AlnisM/1/base
2025-12-04T12:26:06.7585740Z  * [new branch]                gh/AlnisM/1/head        -> origin/gh/AlnisM/1/head
2025-12-04T12:26:06.7585923Z  * [new branch]                gh/EikanWang/67/base    -> origin/gh/EikanWang/67/base
2025-12-04T12:26:06.7586122Z  * [new branch]                gh/EikanWang/67/head    -> origin/gh/EikanWang/67/head
2025-12-04T12:26:06.7586315Z  * [new branch]                gh/Gasoonjia/1/base     -> origin/gh/Gasoonjia/1/base
2025-12-04T12:26:06.7586552Z  * [new branch]                gh/Gasoonjia/1/head     -> origin/gh/Gasoonjia/1/head
2025-12-04T12:26:06.7586814Z  * [new branch]                gh/H-Huang/131/base     -> origin/gh/H-Huang/131/base
2025-12-04T12:26:06.7587006Z  * [new branch]                gh/H-Huang/131/head     -> origin/gh/H-Huang/131/head
2025-12-04T12:26:06.7587193Z  * [new branch]                gh/H-Huang/131/orig     -> origin/gh/H-Huang/131/orig
2025-12-04T12:26:06.7587377Z  * [new branch]                gh/H-Huang/132/base     -> origin/gh/H-Huang/132/base
2025-12-04T12:26:06.7587569Z  * [new branch]                gh/H-Huang/132/head     -> origin/gh/H-Huang/132/head
2025-12-04T12:26:06.7587748Z  * [new branch]                gh/H-Huang/132/orig     -> origin/gh/H-Huang/132/orig
2025-12-04T12:26:06.7587931Z  * [new branch]                gh/H-Huang/180/base     -> origin/gh/H-Huang/180/base
2025-12-04T12:26:06.7588113Z  * [new branch]                gh/H-Huang/180/head     -> origin/gh/H-Huang/180/head
2025-12-04T12:26:06.7588291Z  * [new branch]                gh/H-Huang/180/orig     -> origin/gh/H-Huang/180/orig
2025-12-04T12:26:06.7588483Z  * [new branch]                gh/H-Huang/182/base     -> origin/gh/H-Huang/182/base
2025-12-04T12:26:06.7588663Z  * [new branch]                gh/H-Huang/182/head     -> origin/gh/H-Huang/182/head
2025-12-04T12:26:06.7588849Z  * [new branch]                gh/H-Huang/182/orig     -> origin/gh/H-Huang/182/orig
2025-12-04T12:26:06.7589030Z  * [new branch]                gh/H-Huang/226/base     -> origin/gh/H-Huang/226/base
2025-12-04T12:26:06.7589211Z  * [new branch]                gh/H-Huang/226/head     -> origin/gh/H-Huang/226/head
2025-12-04T12:26:06.7589399Z  * [new branch]                gh/H-Huang/226/orig     -> origin/gh/H-Huang/226/orig
2025-12-04T12:26:06.7589581Z  * [new branch]                gh/H-Huang/228/base     -> origin/gh/H-Huang/228/base
2025-12-04T12:26:06.7589760Z  * [new branch]                gh/H-Huang/228/head     -> origin/gh/H-Huang/228/head
2025-12-04T12:26:06.7589945Z  * [new branch]                gh/H-Huang/228/orig     -> origin/gh/H-Huang/228/orig
2025-12-04T12:26:06.7590154Z  * [new branch]                gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base
2025-12-04T12:26:06.7590360Z  * [new branch]                gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head
2025-12-04T12:26:06.7590566Z  * [new branch]                gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig
2025-12-04T12:26:06.7590990Z  * [new branch]                gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base
2025-12-04T12:26:06.7591195Z  * [new branch]                gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head
2025-12-04T12:26:06.7591398Z  * [new branch]                gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig
2025-12-04T12:26:06.7591600Z  * [new branch]                gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base
2025-12-04T12:26:06.7591797Z  * [new branch]                gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head
2025-12-04T12:26:06.7592052Z  * [new branch]                gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig
2025-12-04T12:26:06.7592258Z  * [new branch]                gh/IvanKobzarev/162/base -> origin/gh/IvanKobzarev/162/base
2025-12-04T12:26:06.7592460Z  * [new branch]                gh/IvanKobzarev/162/head -> origin/gh/IvanKobzarev/162/head
2025-12-04T12:26:06.7592693Z  * [new branch]                gh/IvanKobzarev/162/orig -> origin/gh/IvanKobzarev/162/orig
2025-12-04T12:26:06.7592901Z  * [new branch]                gh/IvanKobzarev/163/base -> origin/gh/IvanKobzarev/163/base
2025-12-04T12:26:06.7593102Z  * [new branch]                gh/IvanKobzarev/163/head -> origin/gh/IvanKobzarev/163/head
2025-12-04T12:26:06.7593305Z  * [new branch]                gh/IvanKobzarev/163/orig -> origin/gh/IvanKobzarev/163/orig
2025-12-04T12:26:06.7593511Z  * [new branch]                gh/IvanKobzarev/166/base -> origin/gh/IvanKobzarev/166/base
2025-12-04T12:26:06.7593716Z  * [new branch]                gh/IvanKobzarev/166/head -> origin/gh/IvanKobzarev/166/head
2025-12-04T12:26:06.7593920Z  * [new branch]                gh/IvanKobzarev/166/orig -> origin/gh/IvanKobzarev/166/orig
2025-12-04T12:26:06.7594124Z  * [new branch]                gh/IvanKobzarev/167/base -> origin/gh/IvanKobzarev/167/base
2025-12-04T12:26:06.7594327Z  * [new branch]                gh/IvanKobzarev/167/head -> origin/gh/IvanKobzarev/167/head
2025-12-04T12:26:06.7594531Z  * [new branch]                gh/IvanKobzarev/167/orig -> origin/gh/IvanKobzarev/167/orig
2025-12-04T12:26:06.7594737Z  * [new branch]                gh/IvanKobzarev/168/base -> origin/gh/IvanKobzarev/168/base
2025-12-04T12:26:06.7594941Z  * [new branch]                gh/IvanKobzarev/168/head -> origin/gh/IvanKobzarev/168/head
2025-12-04T12:26:06.7595144Z  * [new branch]                gh/IvanKobzarev/168/orig -> origin/gh/IvanKobzarev/168/orig
2025-12-04T12:26:06.7595343Z  * [new branch]                gh/IvanKobzarev/169/base -> origin/gh/IvanKobzarev/169/base
2025-12-04T12:26:06.7595552Z  * [new branch]                gh/IvanKobzarev/169/head -> origin/gh/IvanKobzarev/169/head
2025-12-04T12:26:06.7595755Z  * [new branch]                gh/IvanKobzarev/169/orig -> origin/gh/IvanKobzarev/169/orig
2025-12-04T12:26:06.7595954Z  * [new branch]                gh/IvanKobzarev/170/base -> origin/gh/IvanKobzarev/170/base
2025-12-04T12:26:06.7596160Z  * [new branch]                gh/IvanKobzarev/170/head -> origin/gh/IvanKobzarev/170/head
2025-12-04T12:26:06.7596363Z  * [new branch]                gh/IvanKobzarev/170/orig -> origin/gh/IvanKobzarev/170/orig
2025-12-04T12:26:06.7596568Z  * [new branch]                gh/IvanKobzarev/171/base -> origin/gh/IvanKobzarev/171/base
2025-12-04T12:26:06.7596808Z  * [new branch]                gh/IvanKobzarev/171/head -> origin/gh/IvanKobzarev/171/head
2025-12-04T12:26:06.7597015Z  * [new branch]                gh/IvanKobzarev/171/orig -> origin/gh/IvanKobzarev/171/orig
2025-12-04T12:26:06.7597215Z  * [new branch]                gh/IvanKobzarev/172/base -> origin/gh/IvanKobzarev/172/base
2025-12-04T12:26:06.7597426Z  * [new branch]                gh/IvanKobzarev/172/head -> origin/gh/IvanKobzarev/172/head
2025-12-04T12:26:06.7597630Z  * [new branch]                gh/IvanKobzarev/172/orig -> origin/gh/IvanKobzarev/172/orig
2025-12-04T12:26:06.7597834Z  * [new branch]                gh/IvanKobzarev/173/base -> origin/gh/IvanKobzarev/173/base
2025-12-04T12:26:06.7598042Z  * [new branch]                gh/IvanKobzarev/173/head -> origin/gh/IvanKobzarev/173/head
2025-12-04T12:26:06.7598252Z  * [new branch]                gh/IvanKobzarev/173/orig -> origin/gh/IvanKobzarev/173/orig
2025-12-04T12:26:06.7598455Z  * [new branch]                gh/IvanKobzarev/174/base -> origin/gh/IvanKobzarev/174/base
2025-12-04T12:26:06.7598658Z  * [new branch]                gh/IvanKobzarev/174/head -> origin/gh/IvanKobzarev/174/head
2025-12-04T12:26:06.7598862Z  * [new branch]                gh/IvanKobzarev/174/orig -> origin/gh/IvanKobzarev/174/orig
2025-12-04T12:26:06.7599108Z  * [new branch]                gh/IvanKobzarev/175/base -> origin/gh/IvanKobzarev/175/base
2025-12-04T12:26:06.7599316Z  * [new branch]                gh/IvanKobzarev/175/head -> origin/gh/IvanKobzarev/175/head
2025-12-04T12:26:06.7599520Z  * [new branch]                gh/IvanKobzarev/175/orig -> origin/gh/IvanKobzarev/175/orig
2025-12-04T12:26:06.7599752Z  * [new branch]                gh/IvanKobzarev/176/base -> origin/gh/IvanKobzarev/176/base
2025-12-04T12:26:06.7599954Z  * [new branch]                gh/IvanKobzarev/176/head -> origin/gh/IvanKobzarev/176/head
2025-12-04T12:26:06.7600163Z  * [new branch]                gh/IvanKobzarev/176/orig -> origin/gh/IvanKobzarev/176/orig
2025-12-04T12:26:06.7600363Z  * [new branch]                gh/IvanKobzarev/177/base -> origin/gh/IvanKobzarev/177/base
2025-12-04T12:26:06.7600567Z  * [new branch]                gh/IvanKobzarev/177/head -> origin/gh/IvanKobzarev/177/head
2025-12-04T12:26:06.7600769Z  * [new branch]                gh/IvanKobzarev/177/orig -> origin/gh/IvanKobzarev/177/orig
2025-12-04T12:26:06.7600973Z  * [new branch]                gh/IvanKobzarev/178/base -> origin/gh/IvanKobzarev/178/base
2025-12-04T12:26:06.7601176Z  * [new branch]                gh/IvanKobzarev/178/head -> origin/gh/IvanKobzarev/178/head
2025-12-04T12:26:06.7601378Z  * [new branch]                gh/IvanKobzarev/178/orig -> origin/gh/IvanKobzarev/178/orig
2025-12-04T12:26:06.7601585Z  * [new branch]                gh/IvanKobzarev/179/base -> origin/gh/IvanKobzarev/179/base
2025-12-04T12:26:06.7601791Z  * [new branch]                gh/IvanKobzarev/179/head -> origin/gh/IvanKobzarev/179/head
2025-12-04T12:26:06.7601994Z  * [new branch]                gh/IvanKobzarev/179/orig -> origin/gh/IvanKobzarev/179/orig
2025-12-04T12:26:06.7602202Z  * [new branch]                gh/IvanKobzarev/180/base -> origin/gh/IvanKobzarev/180/base
2025-12-04T12:26:06.7602406Z  * [new branch]                gh/IvanKobzarev/180/head -> origin/gh/IvanKobzarev/180/head
2025-12-04T12:26:06.7602611Z  * [new branch]                gh/IvanKobzarev/180/orig -> origin/gh/IvanKobzarev/180/orig
2025-12-04T12:26:06.7602816Z  * [new branch]                gh/IvanKobzarev/181/base -> origin/gh/IvanKobzarev/181/base
2025-12-04T12:26:06.7603021Z  * [new branch]                gh/IvanKobzarev/181/head -> origin/gh/IvanKobzarev/181/head
2025-12-04T12:26:06.7603226Z  * [new branch]                gh/IvanKobzarev/181/orig -> origin/gh/IvanKobzarev/181/orig
2025-12-04T12:26:06.7603434Z  * [new branch]                gh/IvanKobzarev/182/base -> origin/gh/IvanKobzarev/182/base
2025-12-04T12:26:06.7603638Z  * [new branch]                gh/IvanKobzarev/182/head -> origin/gh/IvanKobzarev/182/head
2025-12-04T12:26:06.7603840Z  * [new branch]                gh/IvanKobzarev/182/orig -> origin/gh/IvanKobzarev/182/orig
2025-12-04T12:26:06.7604043Z  * [new branch]                gh/IvanKobzarev/183/base -> origin/gh/IvanKobzarev/183/base
2025-12-04T12:26:06.7604251Z  * [new branch]                gh/IvanKobzarev/183/head -> origin/gh/IvanKobzarev/183/head
2025-12-04T12:26:06.7604451Z  * [new branch]                gh/IvanKobzarev/183/orig -> origin/gh/IvanKobzarev/183/orig
2025-12-04T12:26:06.7604655Z  * [new branch]                gh/IvanKobzarev/184/base -> origin/gh/IvanKobzarev/184/base
2025-12-04T12:26:06.7604866Z  * [new branch]                gh/IvanKobzarev/184/head -> origin/gh/IvanKobzarev/184/head
2025-12-04T12:26:06.7605066Z  * [new branch]                gh/IvanKobzarev/184/orig -> origin/gh/IvanKobzarev/184/orig
2025-12-04T12:26:06.7605272Z  * [new branch]                gh/NikhilAPatel/1/base  -> origin/gh/NikhilAPatel/1/base
2025-12-04T12:26:06.7605479Z  * [new branch]                gh/NikhilAPatel/1/head  -> origin/gh/NikhilAPatel/1/head
2025-12-04T12:26:06.7605677Z  * [new branch]                gh/NikhilAPatel/2/base  -> origin/gh/NikhilAPatel/2/base
2025-12-04T12:26:06.7605878Z  * [new branch]                gh/NikhilAPatel/2/head  -> origin/gh/NikhilAPatel/2/head
2025-12-04T12:26:06.7606115Z  * [new branch]                gh/NikhilAPatel/4/base  -> origin/gh/NikhilAPatel/4/base
2025-12-04T12:26:06.7606313Z  * [new branch]                gh/NikhilAPatel/4/head  -> origin/gh/NikhilAPatel/4/head
2025-12-04T12:26:06.7606514Z  * [new branch]                gh/NikhilAPatel/5/base  -> origin/gh/NikhilAPatel/5/base
2025-12-04T12:26:06.7606791Z  * [new branch]                gh/NikhilAPatel/5/head  -> origin/gh/NikhilAPatel/5/head
2025-12-04T12:26:06.7606993Z  * [new branch]                gh/NikhilAPatel/5/orig  -> origin/gh/NikhilAPatel/5/orig
2025-12-04T12:26:06.7607188Z  * [new branch]                gh/PaliC/17/base        -> origin/gh/PaliC/17/base
2025-12-04T12:26:06.7607369Z  * [new branch]                gh/PaliC/17/head        -> origin/gh/PaliC/17/head
2025-12-04T12:26:06.7607551Z  * [new branch]                gh/PaliC/17/orig        -> origin/gh/PaliC/17/orig
2025-12-04T12:26:06.7607739Z  * [new branch]                gh/PaliC/18/base        -> origin/gh/PaliC/18/base
2025-12-04T12:26:06.7607916Z  * [new branch]                gh/PaliC/18/head        -> origin/gh/PaliC/18/head
2025-12-04T12:26:06.7608096Z  * [new branch]                gh/PaliC/18/orig        -> origin/gh/PaliC/18/orig
2025-12-04T12:26:06.7608273Z  * [new branch]                gh/PaliC/20/base        -> origin/gh/PaliC/20/base
2025-12-04T12:26:06.7608449Z  * [new branch]                gh/PaliC/20/head        -> origin/gh/PaliC/20/head
2025-12-04T12:26:06.7608627Z  * [new branch]                gh/PaliC/20/orig        -> origin/gh/PaliC/20/orig
2025-12-04T12:26:06.7608810Z  * [new branch]                gh/PaliC/21/base        -> origin/gh/PaliC/21/base
2025-12-04T12:26:06.7608984Z  * [new branch]                gh/PaliC/21/head        -> origin/gh/PaliC/21/head
2025-12-04T12:26:06.7609161Z  * [new branch]                gh/PaliC/21/orig        -> origin/gh/PaliC/21/orig
2025-12-04T12:26:06.7609340Z  * [new branch]                gh/PaliC/23/base        -> origin/gh/PaliC/23/base
2025-12-04T12:26:06.7609520Z  * [new branch]                gh/PaliC/23/head        -> origin/gh/PaliC/23/head
2025-12-04T12:26:06.7609699Z  * [new branch]                gh/PaliC/23/orig        -> origin/gh/PaliC/23/orig
2025-12-04T12:26:06.7609880Z  * [new branch]                gh/PaliC/24/base        -> origin/gh/PaliC/24/base
2025-12-04T12:26:06.7610056Z  * [new branch]                gh/PaliC/24/head        -> origin/gh/PaliC/24/head
2025-12-04T12:26:06.7610235Z  * [new branch]                gh/PaliC/24/orig        -> origin/gh/PaliC/24/orig
2025-12-04T12:26:06.7610413Z  * [new branch]                gh/PaliC/25/head        -> origin/gh/PaliC/25/head
2025-12-04T12:26:06.7610594Z  * [new branch]                gh/PaliC/25/next        -> origin/gh/PaliC/25/next
2025-12-04T12:26:06.7610775Z  * [new branch]                gh/PaliC/25/orig        -> origin/gh/PaliC/25/orig
2025-12-04T12:26:06.7610951Z  * [new branch]                gh/PaliC/26/head        -> origin/gh/PaliC/26/head
2025-12-04T12:26:06.7611130Z  * [new branch]                gh/PaliC/26/next        -> origin/gh/PaliC/26/next
2025-12-04T12:26:06.7611311Z  * [new branch]                gh/PaliC/26/orig        -> origin/gh/PaliC/26/orig
2025-12-04T12:26:06.7611490Z  * [new branch]                gh/PaliC/27/next        -> origin/gh/PaliC/27/next
2025-12-04T12:26:06.7611673Z  * [new branch]                gh/PaliC/28/head        -> origin/gh/PaliC/28/head
2025-12-04T12:26:06.7611852Z  * [new branch]                gh/PaliC/28/next        -> origin/gh/PaliC/28/next
2025-12-04T12:26:06.7612028Z  * [new branch]                gh/PaliC/28/orig        -> origin/gh/PaliC/28/orig
2025-12-04T12:26:06.7612209Z  * [new branch]                gh/PaliC/29/head        -> origin/gh/PaliC/29/head
2025-12-04T12:26:06.7612394Z  * [new branch]                gh/PaliC/29/next        -> origin/gh/PaliC/29/next
2025-12-04T12:26:06.7612569Z  * [new branch]                gh/PaliC/29/orig        -> origin/gh/PaliC/29/orig
2025-12-04T12:26:06.7612749Z  * [new branch]                gh/PaliC/30/head        -> origin/gh/PaliC/30/head
2025-12-04T12:26:06.7612967Z  * [new branch]                gh/PaliC/30/next        -> origin/gh/PaliC/30/next
2025-12-04T12:26:06.7613144Z  * [new branch]                gh/PaliC/30/orig        -> origin/gh/PaliC/30/orig
2025-12-04T12:26:06.7613326Z  * [new branch]                gh/PaliC/31/head        -> origin/gh/PaliC/31/head
2025-12-04T12:26:06.7613532Z  * [new branch]                gh/PaliC/31/next        -> origin/gh/PaliC/31/next
2025-12-04T12:26:06.7613707Z  * [new branch]                gh/PaliC/31/orig        -> origin/gh/PaliC/31/orig
2025-12-04T12:26:06.7613900Z  * [new branch]                gh/PaulZhang12/25/base  -> origin/gh/PaulZhang12/25/base
2025-12-04T12:26:06.7614104Z  * [new branch]                gh/PaulZhang12/25/head  -> origin/gh/PaulZhang12/25/head
2025-12-04T12:26:06.7614298Z  * [new branch]                gh/PaulZhang12/25/orig  -> origin/gh/PaulZhang12/25/orig
2025-12-04T12:26:06.7614497Z  * [new branch]                gh/PaulZhang12/28/base  -> origin/gh/PaulZhang12/28/base
2025-12-04T12:26:06.7614691Z  * [new branch]                gh/PaulZhang12/28/head  -> origin/gh/PaulZhang12/28/head
2025-12-04T12:26:06.7614889Z  * [new branch]                gh/PaulZhang12/28/orig  -> origin/gh/PaulZhang12/28/orig
2025-12-04T12:26:06.7615092Z  * [new branch]                gh/PaulZhang12/31/base  -> origin/gh/PaulZhang12/31/base
2025-12-04T12:26:06.7615290Z  * [new branch]                gh/PaulZhang12/31/head  -> origin/gh/PaulZhang12/31/head
2025-12-04T12:26:06.7615487Z  * [new branch]                gh/PaulZhang12/31/orig  -> origin/gh/PaulZhang12/31/orig
2025-12-04T12:26:06.7615684Z  * [new branch]                gh/PaulZhang12/37/base  -> origin/gh/PaulZhang12/37/base
2025-12-04T12:26:06.7615875Z  * [new branch]                gh/PaulZhang12/37/head  -> origin/gh/PaulZhang12/37/head
2025-12-04T12:26:06.7616075Z  * [new branch]                gh/PaulZhang12/37/orig  -> origin/gh/PaulZhang12/37/orig
2025-12-04T12:26:06.7616273Z  * [new branch]                gh/PaulZhang12/40/base  -> origin/gh/PaulZhang12/40/base
2025-12-04T12:26:06.7616468Z  * [new branch]                gh/PaulZhang12/40/head  -> origin/gh/PaulZhang12/40/head
2025-12-04T12:26:06.7616664Z  * [new branch]                gh/PaulZhang12/40/orig  -> origin/gh/PaulZhang12/40/orig
2025-12-04T12:26:06.7616920Z  * [new branch]                gh/PaulZhang12/42/base  -> origin/gh/PaulZhang12/42/base
2025-12-04T12:26:06.7617111Z  * [new branch]                gh/PaulZhang12/42/head  -> origin/gh/PaulZhang12/42/head
2025-12-04T12:26:06.7617311Z  * [new branch]                gh/PaulZhang12/43/base  -> origin/gh/PaulZhang12/43/base
2025-12-04T12:26:06.7617505Z  * [new branch]                gh/PaulZhang12/43/head  -> origin/gh/PaulZhang12/43/head
2025-12-04T12:26:06.7617697Z  * [new branch]                gh/PaulZhang12/43/orig  -> origin/gh/PaulZhang12/43/orig
2025-12-04T12:26:06.7617894Z  * [new branch]                gh/PaulZhang12/44/base  -> origin/gh/PaulZhang12/44/base
2025-12-04T12:26:06.7618096Z  * [new branch]                gh/PaulZhang12/44/head  -> origin/gh/PaulZhang12/44/head
2025-12-04T12:26:06.7618288Z  * [new branch]                gh/PaulZhang12/45/base  -> origin/gh/PaulZhang12/45/base
2025-12-04T12:26:06.7618486Z  * [new branch]                gh/PaulZhang12/45/head  -> origin/gh/PaulZhang12/45/head
2025-12-04T12:26:06.7618685Z  * [new branch]                gh/PaulZhang12/45/orig  -> origin/gh/PaulZhang12/45/orig
2025-12-04T12:26:06.7618876Z  * [new branch]                gh/PaulZhang12/46/base  -> origin/gh/PaulZhang12/46/base
2025-12-04T12:26:06.7619072Z  * [new branch]                gh/PaulZhang12/46/head  -> origin/gh/PaulZhang12/46/head
2025-12-04T12:26:06.7619262Z  * [new branch]                gh/PaulZhang12/46/orig  -> origin/gh/PaulZhang12/46/orig
2025-12-04T12:26:06.7619450Z  * [new branch]                gh/PaulZhang12/47/base  -> origin/gh/PaulZhang12/47/base
2025-12-04T12:26:06.7619641Z  * [new branch]                gh/PaulZhang12/47/head  -> origin/gh/PaulZhang12/47/head
2025-12-04T12:26:06.7620007Z  * [new branch]                gh/PaulZhang12/47/orig  -> origin/gh/PaulZhang12/47/orig
2025-12-04T12:26:06.7620195Z  * [new branch]                gh/PaulZhang12/48/base  -> origin/gh/PaulZhang12/48/base
2025-12-04T12:26:06.7620386Z  * [new branch]                gh/PaulZhang12/48/head  -> origin/gh/PaulZhang12/48/head
2025-12-04T12:26:06.7620625Z  * [new branch]                gh/PaulZhang12/48/orig  -> origin/gh/PaulZhang12/48/orig
2025-12-04T12:26:06.7620823Z  * [new branch]                gh/SamGinzburg/11/base  -> origin/gh/SamGinzburg/11/base
2025-12-04T12:26:06.7621017Z  * [new branch]                gh/SamGinzburg/11/head  -> origin/gh/SamGinzburg/11/head
2025-12-04T12:26:06.7621213Z  * [new branch]                gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base
2025-12-04T12:26:06.7621413Z  * [new branch]                gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head
2025-12-04T12:26:06.7621620Z  * [new branch]                gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base
2025-12-04T12:26:06.7621822Z  * [new branch]                gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head
2025-12-04T12:26:06.7622022Z  * [new branch]                gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig
2025-12-04T12:26:06.7622224Z  * [new branch]                gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base
2025-12-04T12:26:06.7622452Z  * [new branch]                gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head
2025-12-04T12:26:06.7622692Z  * [new branch]                gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig
2025-12-04T12:26:06.7623107Z  * [new branch]                gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base
2025-12-04T12:26:06.7623363Z  * [new branch]                gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head
2025-12-04T12:26:06.7623619Z  * [new branch]                gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig
2025-12-04T12:26:06.7623851Z  * [new branch]                gh/SherlockNoMad/15/base -> origin/gh/SherlockNoMad/15/base
2025-12-04T12:26:06.7624106Z  * [new branch]                gh/SherlockNoMad/15/head -> origin/gh/SherlockNoMad/15/head
2025-12-04T12:26:06.7624370Z  * [new branch]                gh/SherlockNoMad/15/orig -> origin/gh/SherlockNoMad/15/orig
2025-12-04T12:26:06.7624610Z  * [new branch]                gh/SherlockNoMad/17/base -> origin/gh/SherlockNoMad/17/base
2025-12-04T12:26:06.7624847Z  * [new branch]                gh/SherlockNoMad/17/head -> origin/gh/SherlockNoMad/17/head
2025-12-04T12:26:06.7625103Z  * [new branch]                gh/SherlockNoMad/17/orig -> origin/gh/SherlockNoMad/17/orig
2025-12-04T12:26:06.7625340Z  * [new branch]                gh/SherlockNoMad/18/base -> origin/gh/SherlockNoMad/18/base
2025-12-04T12:26:06.7625577Z  * [new branch]                gh/SherlockNoMad/18/head -> origin/gh/SherlockNoMad/18/head
2025-12-04T12:26:06.7625829Z  * [new branch]                gh/SherlockNoMad/18/orig -> origin/gh/SherlockNoMad/18/orig
2025-12-04T12:26:06.7626063Z  * [new branch]                gh/SherlockNoMad/19/base -> origin/gh/SherlockNoMad/19/base
2025-12-04T12:26:06.7626298Z  * [new branch]                gh/SherlockNoMad/19/head -> origin/gh/SherlockNoMad/19/head
2025-12-04T12:26:06.7626554Z  * [new branch]                gh/SherlockNoMad/19/orig -> origin/gh/SherlockNoMad/19/orig
2025-12-04T12:26:06.7626844Z  * [new branch]                gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base
2025-12-04T12:26:06.7627084Z  * [new branch]                gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head
2025-12-04T12:26:06.7627355Z  * [new branch]                gh/SherlockNoMad/20/base -> origin/gh/SherlockNoMad/20/base
2025-12-04T12:26:06.7627582Z  * [new branch]                gh/SherlockNoMad/20/head -> origin/gh/SherlockNoMad/20/head
2025-12-04T12:26:06.7627822Z  * [new branch]                gh/SherlockNoMad/20/orig -> origin/gh/SherlockNoMad/20/orig
2025-12-04T12:26:06.7628072Z  * [new branch]                gh/SherlockNoMad/21/base -> origin/gh/SherlockNoMad/21/base
2025-12-04T12:26:06.7628340Z  * [new branch]                gh/SherlockNoMad/21/head -> origin/gh/SherlockNoMad/21/head
2025-12-04T12:26:06.7628576Z  * [new branch]                gh/SherlockNoMad/21/orig -> origin/gh/SherlockNoMad/21/orig
2025-12-04T12:26:06.7653049Z  * [new branch]                gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base
2025-12-04T12:26:06.7653283Z  * [new branch]                gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head
2025-12-04T12:26:06.7653501Z  * [new branch]                gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base
2025-12-04T12:26:06.7653713Z  * [new branch]                gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head
2025-12-04T12:26:06.7653921Z  * [new branch]                gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base
2025-12-04T12:26:06.7654149Z  * [new branch]                gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head
2025-12-04T12:26:06.7654387Z  * [new branch]                gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base
2025-12-04T12:26:06.7654667Z  * [new branch]                gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base
2025-12-04T12:26:06.7654897Z  * [new branch]                gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base
2025-12-04T12:26:06.7655126Z  * [new branch]                gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base
2025-12-04T12:26:06.7655377Z  * [new branch]                gh/StrongerXi/1/base    -> origin/gh/StrongerXi/1/base
2025-12-04T12:26:06.7655581Z  * [new branch]                gh/StrongerXi/1/head    -> origin/gh/StrongerXi/1/head
2025-12-04T12:26:06.7655778Z  * [new branch]                gh/StrongerXi/71/base   -> origin/gh/StrongerXi/71/base
2025-12-04T12:26:06.7655975Z  * [new branch]                gh/StrongerXi/71/head   -> origin/gh/StrongerXi/71/head
2025-12-04T12:26:06.7656167Z  * [new branch]                gh/StrongerXi/72/base   -> origin/gh/StrongerXi/72/base
2025-12-04T12:26:06.7656358Z  * [new branch]                gh/StrongerXi/72/head   -> origin/gh/StrongerXi/72/head
2025-12-04T12:26:06.7656551Z  * [new branch]                gh/StrongerXi/73/base   -> origin/gh/StrongerXi/73/base
2025-12-04T12:26:06.7656794Z  * [new branch]                gh/StrongerXi/73/head   -> origin/gh/StrongerXi/73/head
2025-12-04T12:26:06.7656982Z  * [new branch]                gh/StrongerXi/73/orig   -> origin/gh/StrongerXi/73/orig
2025-12-04T12:26:06.7657177Z  * [new branch]                gh/XilunWu/160/base     -> origin/gh/XilunWu/160/base
2025-12-04T12:26:06.7657365Z  * [new branch]                gh/XilunWu/160/head     -> origin/gh/XilunWu/160/head
2025-12-04T12:26:06.7657568Z  * [new branch]                gh/XilunWu/160/orig     -> origin/gh/XilunWu/160/orig
2025-12-04T12:26:06.7657753Z  * [new branch]                gh/XilunWu/163/base     -> origin/gh/XilunWu/163/base
2025-12-04T12:26:06.7657944Z  * [new branch]                gh/XilunWu/163/head     -> origin/gh/XilunWu/163/head
2025-12-04T12:26:06.7658148Z  * [new branch]                gh/XilunWu/163/orig     -> origin/gh/XilunWu/163/orig
2025-12-04T12:26:06.7658342Z  * [new branch]                gh/XilunWu/168/base     -> origin/gh/XilunWu/168/base
2025-12-04T12:26:06.7658531Z  * [new branch]                gh/XilunWu/168/head     -> origin/gh/XilunWu/168/head
2025-12-04T12:26:06.7658717Z  * [new branch]                gh/XilunWu/168/orig     -> origin/gh/XilunWu/168/orig
2025-12-04T12:26:06.7658905Z  * [new branch]                gh/XilunWu/169/base     -> origin/gh/XilunWu/169/base
2025-12-04T12:26:06.7659086Z  * [new branch]                gh/XilunWu/169/head     -> origin/gh/XilunWu/169/head
2025-12-04T12:26:06.7659269Z  * [new branch]                gh/XilunWu/169/orig     -> origin/gh/XilunWu/169/orig
2025-12-04T12:26:06.7659461Z  * [new branch]                gh/XilunWu/170/base     -> origin/gh/XilunWu/170/base
2025-12-04T12:26:06.7659655Z  * [new branch]                gh/XilunWu/170/head     -> origin/gh/XilunWu/170/head
2025-12-04T12:26:06.7659943Z  * [new branch]                gh/XilunWu/170/orig     -> origin/gh/XilunWu/170/orig
2025-12-04T12:26:06.7660138Z  * [new branch]                gh/XilunWu/171/base     -> origin/gh/XilunWu/171/base
2025-12-04T12:26:06.7660353Z  * [new branch]                gh/XilunWu/171/head     -> origin/gh/XilunWu/171/head
2025-12-04T12:26:06.7660535Z  * [new branch]                gh/XilunWu/171/orig     -> origin/gh/XilunWu/171/orig
2025-12-04T12:26:06.7660721Z  * [new branch]                gh/XilunWu/173/base     -> origin/gh/XilunWu/173/base
2025-12-04T12:26:06.7660903Z  * [new branch]                gh/XilunWu/173/head     -> origin/gh/XilunWu/173/head
2025-12-04T12:26:06.7661085Z  * [new branch]                gh/XilunWu/173/orig     -> origin/gh/XilunWu/173/orig
2025-12-04T12:26:06.7661268Z  * [new branch]                gh/XilunWu/175/base     -> origin/gh/XilunWu/175/base
2025-12-04T12:26:06.7661446Z  * [new branch]                gh/XilunWu/175/head     -> origin/gh/XilunWu/175/head
2025-12-04T12:26:06.7661632Z  * [new branch]                gh/XilunWu/175/orig     -> origin/gh/XilunWu/175/orig
2025-12-04T12:26:06.7661817Z  * [new branch]                gh/XilunWu/176/base     -> origin/gh/XilunWu/176/base
2025-12-04T12:26:06.7662001Z  * [new branch]                gh/XilunWu/176/head     -> origin/gh/XilunWu/176/head
2025-12-04T12:26:06.7662185Z  * [new branch]                gh/XilunWu/176/orig     -> origin/gh/XilunWu/176/orig
2025-12-04T12:26:06.7662368Z  * [new branch]                gh/XuehaiPan/14/base    -> origin/gh/XuehaiPan/14/base
2025-12-04T12:26:06.7662558Z  * [new branch]                gh/XuehaiPan/14/head    -> origin/gh/XuehaiPan/14/head
2025-12-04T12:26:06.7662748Z  * [new branch]                gh/XuehaiPan/14/orig    -> origin/gh/XuehaiPan/14/orig
2025-12-04T12:26:06.7662934Z  * [new branch]                gh/XuehaiPan/179/base   -> origin/gh/XuehaiPan/179/base
2025-12-04T12:26:06.7663126Z  * [new branch]                gh/XuehaiPan/179/head   -> origin/gh/XuehaiPan/179/head
2025-12-04T12:26:06.7663324Z  * [new branch]                gh/XuehaiPan/179/orig   -> origin/gh/XuehaiPan/179/orig
2025-12-04T12:26:06.7663510Z  * [new branch]                gh/XuehaiPan/249/base   -> origin/gh/XuehaiPan/249/base
2025-12-04T12:26:06.7663706Z  * [new branch]                gh/XuehaiPan/249/head   -> origin/gh/XuehaiPan/249/head
2025-12-04T12:26:06.7663898Z  * [new branch]                gh/XuehaiPan/249/orig   -> origin/gh/XuehaiPan/249/orig
2025-12-04T12:26:06.7664085Z  * [new branch]                gh/XuehaiPan/253/base   -> origin/gh/XuehaiPan/253/base
2025-12-04T12:26:06.7664274Z  * [new branch]                gh/XuehaiPan/253/head   -> origin/gh/XuehaiPan/253/head
2025-12-04T12:26:06.7664464Z  * [new branch]                gh/XuehaiPan/253/orig   -> origin/gh/XuehaiPan/253/orig
2025-12-04T12:26:06.7664648Z  * [new branch]                gh/XuehaiPan/254/base   -> origin/gh/XuehaiPan/254/base
2025-12-04T12:26:06.7664838Z  * [new branch]                gh/XuehaiPan/254/head   -> origin/gh/XuehaiPan/254/head
2025-12-04T12:26:06.7665026Z  * [new branch]                gh/XuehaiPan/254/orig   -> origin/gh/XuehaiPan/254/orig
2025-12-04T12:26:06.7665207Z  * [new branch]                gh/XuehaiPan/255/base   -> origin/gh/XuehaiPan/255/base
2025-12-04T12:26:06.7665397Z  * [new branch]                gh/XuehaiPan/255/head   -> origin/gh/XuehaiPan/255/head
2025-12-04T12:26:06.7665587Z  * [new branch]                gh/XuehaiPan/255/orig   -> origin/gh/XuehaiPan/255/orig
2025-12-04T12:26:06.7665773Z  * [new branch]                gh/XuehaiPan/271/base   -> origin/gh/XuehaiPan/271/base
2025-12-04T12:26:06.7665962Z  * [new branch]                gh/XuehaiPan/271/head   -> origin/gh/XuehaiPan/271/head
2025-12-04T12:26:06.7666151Z  * [new branch]                gh/XuehaiPan/271/orig   -> origin/gh/XuehaiPan/271/orig
2025-12-04T12:26:06.7666337Z  * [new branch]                gh/XuehaiPan/343/base   -> origin/gh/XuehaiPan/343/base
2025-12-04T12:26:06.7666554Z  * [new branch]                gh/XuehaiPan/343/head   -> origin/gh/XuehaiPan/343/head
2025-12-04T12:26:06.7666798Z  * [new branch]                gh/XuehaiPan/343/orig   -> origin/gh/XuehaiPan/343/orig
2025-12-04T12:26:06.7666985Z  * [new branch]                gh/XuehaiPan/347/base   -> origin/gh/XuehaiPan/347/base
2025-12-04T12:26:06.7667249Z  * [new branch]                gh/XuehaiPan/347/head   -> origin/gh/XuehaiPan/347/head
2025-12-04T12:26:06.7667434Z  * [new branch]                gh/XuehaiPan/347/orig   -> origin/gh/XuehaiPan/347/orig
2025-12-04T12:26:06.7667620Z  * [new branch]                gh/XuehaiPan/348/base   -> origin/gh/XuehaiPan/348/base
2025-12-04T12:26:06.7667807Z  * [new branch]                gh/XuehaiPan/348/head   -> origin/gh/XuehaiPan/348/head
2025-12-04T12:26:06.7667992Z  * [new branch]                gh/XuehaiPan/348/orig   -> origin/gh/XuehaiPan/348/orig
2025-12-04T12:26:06.7668180Z  * [new branch]                gh/XuehaiPan/350/base   -> origin/gh/XuehaiPan/350/base
2025-12-04T12:26:06.7668371Z  * [new branch]                gh/XuehaiPan/350/head   -> origin/gh/XuehaiPan/350/head
2025-12-04T12:26:06.7668557Z  * [new branch]                gh/XuehaiPan/350/orig   -> origin/gh/XuehaiPan/350/orig
2025-12-04T12:26:06.7668743Z  * [new branch]                gh/XuehaiPan/365/base   -> origin/gh/XuehaiPan/365/base
2025-12-04T12:26:06.7668933Z  * [new branch]                gh/XuehaiPan/365/head   -> origin/gh/XuehaiPan/365/head
2025-12-04T12:26:06.7669117Z  * [new branch]                gh/XuehaiPan/365/orig   -> origin/gh/XuehaiPan/365/orig
2025-12-04T12:26:06.7669304Z  * [new branch]                gh/XuehaiPan/366/base   -> origin/gh/XuehaiPan/366/base
2025-12-04T12:26:06.7669492Z  * [new branch]                gh/XuehaiPan/366/head   -> origin/gh/XuehaiPan/366/head
2025-12-04T12:26:06.7669678Z  * [new branch]                gh/XuehaiPan/370/base   -> origin/gh/XuehaiPan/370/base
2025-12-04T12:26:06.7669866Z  * [new branch]                gh/XuehaiPan/370/head   -> origin/gh/XuehaiPan/370/head
2025-12-04T12:26:06.7670054Z  * [new branch]                gh/XuehaiPan/370/orig   -> origin/gh/XuehaiPan/370/orig
2025-12-04T12:26:06.7670238Z  * [new branch]                gh/XuehaiPan/390/base   -> origin/gh/XuehaiPan/390/base
2025-12-04T12:26:06.7670423Z  * [new branch]                gh/XuehaiPan/390/head   -> origin/gh/XuehaiPan/390/head
2025-12-04T12:26:06.7670613Z  * [new branch]                gh/XuehaiPan/390/orig   -> origin/gh/XuehaiPan/390/orig
2025-12-04T12:26:06.7670796Z  * [new branch]                gh/XuehaiPan/391/base   -> origin/gh/XuehaiPan/391/base
2025-12-04T12:26:06.7670983Z  * [new branch]                gh/XuehaiPan/391/head   -> origin/gh/XuehaiPan/391/head
2025-12-04T12:26:06.7671172Z  * [new branch]                gh/XuehaiPan/391/orig   -> origin/gh/XuehaiPan/391/orig
2025-12-04T12:26:06.7671360Z  * [new branch]                gh/XuehaiPan/392/base   -> origin/gh/XuehaiPan/392/base
2025-12-04T12:26:06.7671550Z  * [new branch]                gh/XuehaiPan/392/head   -> origin/gh/XuehaiPan/392/head
2025-12-04T12:26:06.7671738Z  * [new branch]                gh/XuehaiPan/392/orig   -> origin/gh/XuehaiPan/392/orig
2025-12-04T12:26:06.7671928Z  * [new branch]                gh/XuehaiPan/394/base   -> origin/gh/XuehaiPan/394/base
2025-12-04T12:26:06.7672123Z  * [new branch]                gh/XuehaiPan/394/head   -> origin/gh/XuehaiPan/394/head
2025-12-04T12:26:06.7672312Z  * [new branch]                gh/XuehaiPan/394/orig   -> origin/gh/XuehaiPan/394/orig
2025-12-04T12:26:06.7672500Z  * [new branch]                gh/XuehaiPan/397/base   -> origin/gh/XuehaiPan/397/base
2025-12-04T12:26:06.7672688Z  * [new branch]                gh/XuehaiPan/397/head   -> origin/gh/XuehaiPan/397/head
2025-12-04T12:26:06.7672872Z  * [new branch]                gh/XuehaiPan/397/orig   -> origin/gh/XuehaiPan/397/orig
2025-12-04T12:26:06.7673060Z  * [new branch]                gh/XuehaiPan/398/base   -> origin/gh/XuehaiPan/398/base
2025-12-04T12:26:06.7673249Z  * [new branch]                gh/XuehaiPan/398/head   -> origin/gh/XuehaiPan/398/head
2025-12-04T12:26:06.7673476Z  * [new branch]                gh/XuehaiPan/398/orig   -> origin/gh/XuehaiPan/398/orig
2025-12-04T12:26:06.7673664Z  * [new branch]                gh/XuehaiPan/399/base   -> origin/gh/XuehaiPan/399/base
2025-12-04T12:26:06.7673885Z  * [new branch]                gh/XuehaiPan/399/head   -> origin/gh/XuehaiPan/399/head
2025-12-04T12:26:06.7674073Z  * [new branch]                gh/XuehaiPan/399/orig   -> origin/gh/XuehaiPan/399/orig
2025-12-04T12:26:06.7674261Z  * [new branch]                gh/XuehaiPan/400/base   -> origin/gh/XuehaiPan/400/base
2025-12-04T12:26:06.7674448Z  * [new branch]                gh/XuehaiPan/400/head   -> origin/gh/XuehaiPan/400/head
2025-12-04T12:26:06.7674633Z  * [new branch]                gh/XuehaiPan/400/orig   -> origin/gh/XuehaiPan/400/orig
2025-12-04T12:26:06.7674829Z  * [new branch]                gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base
2025-12-04T12:26:06.7675026Z  * [new branch]                gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head
2025-12-04T12:26:06.7675222Z  * [new branch]                gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig
2025-12-04T12:26:06.7675415Z  * [new branch]                gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base
2025-12-04T12:26:06.7675608Z  * [new branch]                gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head
2025-12-04T12:26:06.7675798Z  * [new branch]                gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base
2025-12-04T12:26:06.7675991Z  * [new branch]                gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head
2025-12-04T12:26:06.7676179Z  * [new branch]                gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base
2025-12-04T12:26:06.7676369Z  * [new branch]                gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head
2025-12-04T12:26:06.7676556Z  * [new branch]                gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base
2025-12-04T12:26:06.7676780Z  * [new branch]                gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head
2025-12-04T12:26:06.7676973Z  * [new branch]                gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base
2025-12-04T12:26:06.7677169Z  * [new branch]                gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head
2025-12-04T12:26:06.7677363Z  * [new branch]                gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base
2025-12-04T12:26:06.7677555Z  * [new branch]                gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head
2025-12-04T12:26:06.7677749Z  * [new branch]                gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base
2025-12-04T12:26:06.7677939Z  * [new branch]                gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head
2025-12-04T12:26:06.7678131Z  * [new branch]                gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig
2025-12-04T12:26:06.7678326Z  * [new branch]                gh/aakhundov/1/base     -> origin/gh/aakhundov/1/base
2025-12-04T12:26:06.7678513Z  * [new branch]                gh/aakhundov/1/head     -> origin/gh/aakhundov/1/head
2025-12-04T12:26:06.7678696Z  * [new branch]                gh/aakhundov/2/base     -> origin/gh/aakhundov/2/base
2025-12-04T12:26:06.7678880Z  * [new branch]                gh/aakhundov/2/head     -> origin/gh/aakhundov/2/head
2025-12-04T12:26:06.7679064Z  * [new branch]                gh/aditew01/openblas    -> origin/gh/aditew01/openblas
2025-12-04T12:26:06.7679253Z  * [new branch]                gh/aditew01/sbgemm      -> origin/gh/aditew01/sbgemm
2025-12-04T12:26:06.7679438Z  * [new branch]                gh/aditew01/vecbf16     -> origin/gh/aditew01/vecbf16
2025-12-04T12:26:06.7679618Z  * [new branch]                gh/albanD/4/base        -> origin/gh/albanD/4/base
2025-12-04T12:26:06.7679796Z  * [new branch]                gh/albanD/4/head        -> origin/gh/albanD/4/head
2025-12-04T12:26:06.7679966Z  * [new branch]                gh/albanD/4/orig        -> origin/gh/albanD/4/orig
2025-12-04T12:26:06.7680267Z  * [new branch]                gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init
2025-12-04T12:26:06.7680545Z  * [new branch]                gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base
2025-12-04T12:26:06.7680781Z  * [new branch]                gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head
2025-12-04T12:26:06.7680982Z  * [new branch]                gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig
2025-12-04T12:26:06.7681181Z  * [new branch]                gh/alexsamardzic/14/base -> origin/gh/alexsamardzic/14/base
2025-12-04T12:26:06.7681377Z  * [new branch]                gh/alexsamardzic/14/head -> origin/gh/alexsamardzic/14/head
2025-12-04T12:26:06.7681574Z  * [new branch]                gh/alexsamardzic/14/orig -> origin/gh/alexsamardzic/14/orig
2025-12-04T12:26:06.7681773Z  * [new branch]                gh/alexsamardzic/15/base -> origin/gh/alexsamardzic/15/base
2025-12-04T12:26:06.7681969Z  * [new branch]                gh/alexsamardzic/15/head -> origin/gh/alexsamardzic/15/head
2025-12-04T12:26:06.7682168Z  * [new branch]                gh/alexsamardzic/15/orig -> origin/gh/alexsamardzic/15/orig
2025-12-04T12:26:06.7682360Z  * [new branch]                gh/amjames/18/base      -> origin/gh/amjames/18/base
2025-12-04T12:26:06.7682543Z  * [new branch]                gh/amjames/18/head      -> origin/gh/amjames/18/head
2025-12-04T12:26:06.7682721Z  * [new branch]                gh/amjames/18/orig      -> origin/gh/amjames/18/orig
2025-12-04T12:26:06.7682909Z  * [new branch]                gh/andrewor14/35/base   -> origin/gh/andrewor14/35/base
2025-12-04T12:26:06.7683099Z  * [new branch]                gh/andrewor14/35/head   -> origin/gh/andrewor14/35/head
2025-12-04T12:26:06.7683288Z  * [new branch]                gh/andrewor14/35/orig   -> origin/gh/andrewor14/35/orig
2025-12-04T12:26:06.7683473Z  * [new branch]                gh/andrewor14/50/base   -> origin/gh/andrewor14/50/base
2025-12-04T12:26:06.7683660Z  * [new branch]                gh/andrewor14/50/head   -> origin/gh/andrewor14/50/head
2025-12-04T12:26:06.7683845Z  * [new branch]                gh/andrewor14/50/orig   -> origin/gh/andrewor14/50/orig
2025-12-04T12:26:06.7684039Z  * [new branch]                gh/andyanwang/30/base   -> origin/gh/andyanwang/30/base
2025-12-04T12:26:06.7684224Z  * [new branch]                gh/andyanwang/30/orig   -> origin/gh/andyanwang/30/orig
2025-12-04T12:26:06.7684410Z  * [new branch]                gh/andyanwang/31/base   -> origin/gh/andyanwang/31/base
2025-12-04T12:26:06.7684598Z  * [new branch]                gh/andyanwang/31/orig   -> origin/gh/andyanwang/31/orig
2025-12-04T12:26:06.7684785Z  * [new branch]                gh/andyanwang/39/base   -> origin/gh/andyanwang/39/base
2025-12-04T12:26:06.7684970Z  * [new branch]                gh/andyanwang/39/head   -> origin/gh/andyanwang/39/head
2025-12-04T12:26:06.7685160Z  * [new branch]                gh/andyanwang/39/orig   -> origin/gh/andyanwang/39/orig
2025-12-04T12:26:06.7685347Z  * [new branch]                gh/andyanwang/42/base   -> origin/gh/andyanwang/42/base
2025-12-04T12:26:06.7685537Z  * [new branch]                gh/andyanwang/42/head   -> origin/gh/andyanwang/42/head
2025-12-04T12:26:06.7685724Z  * [new branch]                gh/andyanwang/42/orig   -> origin/gh/andyanwang/42/orig
2025-12-04T12:26:06.7685910Z  * [new branch]                gh/andyanwang/45/base   -> origin/gh/andyanwang/45/base
2025-12-04T12:26:06.7686100Z  * [new branch]                gh/andyanwang/45/head   -> origin/gh/andyanwang/45/head
2025-12-04T12:26:06.7686298Z  * [new branch]                gh/andyanwang/45/orig   -> origin/gh/andyanwang/45/orig
2025-12-04T12:26:06.7686485Z  * [new branch]                gh/angelayi/107/base    -> origin/gh/angelayi/107/base
2025-12-04T12:26:06.7686674Z  * [new branch]                gh/angelayi/107/head    -> origin/gh/angelayi/107/head
2025-12-04T12:26:06.7686907Z  * [new branch]                gh/angelayi/114/base    -> origin/gh/angelayi/114/base
2025-12-04T12:26:06.7687129Z  * [new branch]                gh/angelayi/114/head    -> origin/gh/angelayi/114/head
2025-12-04T12:26:06.7687318Z  * [new branch]                gh/angelayi/114/orig    -> origin/gh/angelayi/114/orig
2025-12-04T12:26:06.7687534Z  * [new branch]                gh/angelayi/116/base    -> origin/gh/angelayi/116/base
2025-12-04T12:26:06.7687717Z  * [new branch]                gh/angelayi/116/head    -> origin/gh/angelayi/116/head
2025-12-04T12:26:06.7687902Z  * [new branch]                gh/angelayi/116/orig    -> origin/gh/angelayi/116/orig
2025-12-04T12:26:06.7688083Z  * [new branch]                gh/angelayi/122/base    -> origin/gh/angelayi/122/base
2025-12-04T12:26:06.7688269Z  * [new branch]                gh/angelayi/122/head    -> origin/gh/angelayi/122/head
2025-12-04T12:26:06.7688454Z  * [new branch]                gh/angelayi/122/orig    -> origin/gh/angelayi/122/orig
2025-12-04T12:26:06.7688635Z  * [new branch]                gh/angelayi/124/base    -> origin/gh/angelayi/124/base
2025-12-04T12:26:06.7688827Z  * [new branch]                gh/angelayi/124/head    -> origin/gh/angelayi/124/head
2025-12-04T12:26:06.7689015Z  * [new branch]                gh/angelayi/124/orig    -> origin/gh/angelayi/124/orig
2025-12-04T12:26:06.7689199Z  * [new branch]                gh/angelayi/128/base    -> origin/gh/angelayi/128/base
2025-12-04T12:26:06.7689384Z  * [new branch]                gh/angelayi/128/head    -> origin/gh/angelayi/128/head
2025-12-04T12:26:06.7689571Z  * [new branch]                gh/angelayi/128/orig    -> origin/gh/angelayi/128/orig
2025-12-04T12:26:06.7689753Z  * [new branch]                gh/angelayi/131/base    -> origin/gh/angelayi/131/base
2025-12-04T12:26:06.7689937Z  * [new branch]                gh/angelayi/131/head    -> origin/gh/angelayi/131/head
2025-12-04T12:26:06.7690125Z  * [new branch]                gh/angelayi/131/orig    -> origin/gh/angelayi/131/orig
2025-12-04T12:26:06.7690315Z  * [new branch]                gh/angelayi/132/base    -> origin/gh/angelayi/132/base
2025-12-04T12:26:06.7690503Z  * [new branch]                gh/angelayi/132/head    -> origin/gh/angelayi/132/head
2025-12-04T12:26:06.7690687Z  * [new branch]                gh/angelayi/132/orig    -> origin/gh/angelayi/132/orig
2025-12-04T12:26:06.7690876Z  * [new branch]                gh/angelayi/133/base    -> origin/gh/angelayi/133/base
2025-12-04T12:26:06.7691063Z  * [new branch]                gh/angelayi/133/head    -> origin/gh/angelayi/133/head
2025-12-04T12:26:06.7691253Z  * [new branch]                gh/angelayi/133/orig    -> origin/gh/angelayi/133/orig
2025-12-04T12:26:06.7691435Z  * [new branch]                gh/angelayi/134/base    -> origin/gh/angelayi/134/base
2025-12-04T12:26:06.7691622Z  * [new branch]                gh/angelayi/134/head    -> origin/gh/angelayi/134/head
2025-12-04T12:26:06.7691808Z  * [new branch]                gh/angelayi/134/orig    -> origin/gh/angelayi/134/orig
2025-12-04T12:26:06.7691996Z  * [new branch]                gh/angelayi/135/base    -> origin/gh/angelayi/135/base
2025-12-04T12:26:06.7692184Z  * [new branch]                gh/angelayi/135/head    -> origin/gh/angelayi/135/head
2025-12-04T12:26:06.7692368Z  * [new branch]                gh/angelayi/135/orig    -> origin/gh/angelayi/135/orig
2025-12-04T12:26:06.7692553Z  * [new branch]                gh/angelayi/136/base    -> origin/gh/angelayi/136/base
2025-12-04T12:26:06.7692737Z  * [new branch]                gh/angelayi/136/head    -> origin/gh/angelayi/136/head
2025-12-04T12:26:06.7692919Z  * [new branch]                gh/angelayi/136/orig    -> origin/gh/angelayi/136/orig
2025-12-04T12:26:06.7693106Z  * [new branch]                gh/angelayi/137/base    -> origin/gh/angelayi/137/base
2025-12-04T12:26:06.7693292Z  * [new branch]                gh/angelayi/137/head    -> origin/gh/angelayi/137/head
2025-12-04T12:26:06.7693473Z  * [new branch]                gh/angelayi/137/orig    -> origin/gh/angelayi/137/orig
2025-12-04T12:26:06.7693689Z  * [new branch]                gh/angelayi/138/base    -> origin/gh/angelayi/138/base
2025-12-04T12:26:06.7693875Z  * [new branch]                gh/angelayi/138/head    -> origin/gh/angelayi/138/head
2025-12-04T12:26:06.7694056Z  * [new branch]                gh/angelayi/138/orig    -> origin/gh/angelayi/138/orig
2025-12-04T12:26:06.7694285Z  * [new branch]                gh/angelayi/139/base    -> origin/gh/angelayi/139/base
2025-12-04T12:26:06.7694470Z  * [new branch]                gh/angelayi/139/head    -> origin/gh/angelayi/139/head
2025-12-04T12:26:06.7694655Z  * [new branch]                gh/angelayi/139/orig    -> origin/gh/angelayi/139/orig
2025-12-04T12:26:06.7694839Z  * [new branch]                gh/angelayi/140/base    -> origin/gh/angelayi/140/base
2025-12-04T12:26:06.7695027Z  * [new branch]                gh/angelayi/140/head    -> origin/gh/angelayi/140/head
2025-12-04T12:26:06.7695208Z  * [new branch]                gh/angelayi/140/orig    -> origin/gh/angelayi/140/orig
2025-12-04T12:26:06.7695395Z  * [new branch]                gh/angelayi/141/base    -> origin/gh/angelayi/141/base
2025-12-04T12:26:06.7695580Z  * [new branch]                gh/angelayi/141/head    -> origin/gh/angelayi/141/head
2025-12-04T12:26:06.7695764Z  * [new branch]                gh/angelayi/141/orig    -> origin/gh/angelayi/141/orig
2025-12-04T12:26:06.7695951Z  * [new branch]                gh/angelayi/142/base    -> origin/gh/angelayi/142/base
2025-12-04T12:26:06.7696134Z  * [new branch]                gh/angelayi/142/head    -> origin/gh/angelayi/142/head
2025-12-04T12:26:06.7696315Z  * [new branch]                gh/angelayi/142/orig    -> origin/gh/angelayi/142/orig
2025-12-04T12:26:06.7696498Z  * [new branch]                gh/angelayi/143/base    -> origin/gh/angelayi/143/base
2025-12-04T12:26:06.7696801Z  * [new branch]                gh/angelayi/143/head    -> origin/gh/angelayi/143/head
2025-12-04T12:26:06.7696990Z  * [new branch]                gh/angelayi/143/orig    -> origin/gh/angelayi/143/orig
2025-12-04T12:26:06.7697182Z  * [new branch]                gh/angelayi/144/base    -> origin/gh/angelayi/144/base
2025-12-04T12:26:06.7697364Z  * [new branch]                gh/angelayi/144/head    -> origin/gh/angelayi/144/head
2025-12-04T12:26:06.7697550Z  * [new branch]                gh/angelayi/144/orig    -> origin/gh/angelayi/144/orig
2025-12-04T12:26:06.7697748Z  * [new branch]                gh/anijain2305/753/base -> origin/gh/anijain2305/753/base
2025-12-04T12:26:06.7697940Z  * [new branch]                gh/anijain2305/753/head -> origin/gh/anijain2305/753/head
2025-12-04T12:26:06.7698132Z  * [new branch]                gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig
2025-12-04T12:26:06.7698325Z  * [new branch]                gh/anijain2305/810/base -> origin/gh/anijain2305/810/base
2025-12-04T12:26:06.7698517Z  * [new branch]                gh/anijain2305/810/head -> origin/gh/anijain2305/810/head
2025-12-04T12:26:06.7698706Z  * [new branch]                gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig
2025-12-04T12:26:06.7698901Z  * [new branch]                gh/anijain2305/854/base -> origin/gh/anijain2305/854/base
2025-12-04T12:26:06.7699090Z  * [new branch]                gh/anijain2305/854/head -> origin/gh/anijain2305/854/head
2025-12-04T12:26:06.7699283Z  * [new branch]                gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig
2025-12-04T12:26:06.7699479Z  * [new branch]                gh/anijain2305/864/base -> origin/gh/anijain2305/864/base
2025-12-04T12:26:06.7699665Z  * [new branch]                gh/anijain2305/864/head -> origin/gh/anijain2305/864/head
2025-12-04T12:26:06.7699856Z  * [new branch]                gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig
2025-12-04T12:26:06.7700045Z  * [new branch]                gh/anijain2305/870/base -> origin/gh/anijain2305/870/base
2025-12-04T12:26:06.7700232Z  * [new branch]                gh/anijain2305/870/head -> origin/gh/anijain2305/870/head
2025-12-04T12:26:06.7700469Z  * [new branch]                gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig
2025-12-04T12:26:06.7700660Z  * [new branch]                gh/anijain2305/873/base -> origin/gh/anijain2305/873/base
2025-12-04T12:26:06.7700847Z  * [new branch]                gh/anijain2305/873/head -> origin/gh/anijain2305/873/head
2025-12-04T12:26:06.7701073Z  * [new branch]                gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig
2025-12-04T12:26:06.7701267Z  * [new branch]                gh/anijain2305/894/base -> origin/gh/anijain2305/894/base
2025-12-04T12:26:06.7701454Z  * [new branch]                gh/anijain2305/894/head -> origin/gh/anijain2305/894/head
2025-12-04T12:26:06.7701644Z  * [new branch]                gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig
2025-12-04T12:26:06.7701830Z  * [new branch]                gh/anijain2305/895/base -> origin/gh/anijain2305/895/base
2025-12-04T12:26:06.7702021Z  * [new branch]                gh/anijain2305/895/head -> origin/gh/anijain2305/895/head
2025-12-04T12:26:06.7702217Z  * [new branch]                gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig
2025-12-04T12:26:06.7702405Z  * [new branch]                gh/anijain2305/910/base -> origin/gh/anijain2305/910/base
2025-12-04T12:26:06.7702596Z  * [new branch]                gh/anijain2305/910/head -> origin/gh/anijain2305/910/head
2025-12-04T12:26:06.7702787Z  * [new branch]                gh/anijain2305/910/orig -> origin/gh/anijain2305/910/orig
2025-12-04T12:26:06.7702973Z  * [new branch]                gh/anijain2305/919/base -> origin/gh/anijain2305/919/base
2025-12-04T12:26:06.7703166Z  * [new branch]                gh/anijain2305/919/head -> origin/gh/anijain2305/919/head
2025-12-04T12:26:06.7703355Z  * [new branch]                gh/anijain2305/919/orig -> origin/gh/anijain2305/919/orig
2025-12-04T12:26:06.7703544Z  * [new branch]                gh/anijain2305/922/base -> origin/gh/anijain2305/922/base
2025-12-04T12:26:06.7703732Z  * [new branch]                gh/anijain2305/922/head -> origin/gh/anijain2305/922/head
2025-12-04T12:26:06.7703928Z  * [new branch]                gh/anijain2305/922/orig -> origin/gh/anijain2305/922/orig
2025-12-04T12:26:06.7704113Z  * [new branch]                gh/anijain2305/932/base -> origin/gh/anijain2305/932/base
2025-12-04T12:26:06.7704305Z  * [new branch]                gh/anijain2305/932/head -> origin/gh/anijain2305/932/head
2025-12-04T12:26:06.7704495Z  * [new branch]                gh/anijain2305/932/orig -> origin/gh/anijain2305/932/orig
2025-12-04T12:26:06.7704682Z  * [new branch]                gh/anijain2305/940/base -> origin/gh/anijain2305/940/base
2025-12-04T12:26:06.7704873Z  * [new branch]                gh/anijain2305/940/head -> origin/gh/anijain2305/940/head
2025-12-04T12:26:06.7705066Z  * [new branch]                gh/anijain2305/940/orig -> origin/gh/anijain2305/940/orig
2025-12-04T12:26:06.7705253Z  * [new branch]                gh/anijain2305/941/base -> origin/gh/anijain2305/941/base
2025-12-04T12:26:06.7705443Z  * [new branch]                gh/anijain2305/941/head -> origin/gh/anijain2305/941/head
2025-12-04T12:26:06.7705638Z  * [new branch]                gh/anijain2305/941/orig -> origin/gh/anijain2305/941/orig
2025-12-04T12:26:06.7705827Z  * [new branch]                gh/anijain2305/942/base -> origin/gh/anijain2305/942/base
2025-12-04T12:26:06.7706019Z  * [new branch]                gh/anijain2305/942/head -> origin/gh/anijain2305/942/head
2025-12-04T12:26:06.7706208Z  * [new branch]                gh/anijain2305/942/orig -> origin/gh/anijain2305/942/orig
2025-12-04T12:26:06.7706402Z  * [new branch]                gh/anijain2305/943/base -> origin/gh/anijain2305/943/base
2025-12-04T12:26:06.7706592Z  * [new branch]                gh/anijain2305/943/head -> origin/gh/anijain2305/943/head
2025-12-04T12:26:06.7706814Z  * [new branch]                gh/anijain2305/943/orig -> origin/gh/anijain2305/943/orig
2025-12-04T12:26:06.7707008Z  * [new branch]                gh/anijain2305/944/base -> origin/gh/anijain2305/944/base
2025-12-04T12:26:06.7707229Z  * [new branch]                gh/anijain2305/944/head -> origin/gh/anijain2305/944/head
2025-12-04T12:26:06.7707419Z  * [new branch]                gh/anijain2305/944/orig -> origin/gh/anijain2305/944/orig
2025-12-04T12:26:06.7707608Z  * [new branch]                gh/anijain2305/945/base -> origin/gh/anijain2305/945/base
2025-12-04T12:26:06.7707835Z  * [new branch]                gh/anijain2305/945/head -> origin/gh/anijain2305/945/head
2025-12-04T12:26:06.7708024Z  * [new branch]                gh/anijain2305/945/orig -> origin/gh/anijain2305/945/orig
2025-12-04T12:26:06.7708220Z  * [new branch]                gh/anijain2305/946/base -> origin/gh/anijain2305/946/base
2025-12-04T12:26:06.7708411Z  * [new branch]                gh/anijain2305/946/head -> origin/gh/anijain2305/946/head
2025-12-04T12:26:06.7708598Z  * [new branch]                gh/anijain2305/946/orig -> origin/gh/anijain2305/946/orig
2025-12-04T12:26:06.7708787Z  * [new branch]                gh/anijain2305/947/base -> origin/gh/anijain2305/947/base
2025-12-04T12:26:06.7708979Z  * [new branch]                gh/anijain2305/947/head -> origin/gh/anijain2305/947/head
2025-12-04T12:26:06.7709169Z  * [new branch]                gh/anijain2305/947/orig -> origin/gh/anijain2305/947/orig
2025-12-04T12:26:06.7709363Z  * [new branch]                gh/anijain2305/948/base -> origin/gh/anijain2305/948/base
2025-12-04T12:26:06.7709553Z  * [new branch]                gh/anijain2305/948/head -> origin/gh/anijain2305/948/head
2025-12-04T12:26:06.7709739Z  * [new branch]                gh/anijain2305/948/orig -> origin/gh/anijain2305/948/orig
2025-12-04T12:26:06.7709930Z  * [new branch]                gh/anijain2305/949/base -> origin/gh/anijain2305/949/base
2025-12-04T12:26:06.7710125Z  * [new branch]                gh/anijain2305/949/head -> origin/gh/anijain2305/949/head
2025-12-04T12:26:06.7710316Z  * [new branch]                gh/anijain2305/949/orig -> origin/gh/anijain2305/949/orig
2025-12-04T12:26:06.7710505Z  * [new branch]                gh/anijain2305/950/base -> origin/gh/anijain2305/950/base
2025-12-04T12:26:06.7710693Z  * [new branch]                gh/anijain2305/950/head -> origin/gh/anijain2305/950/head
2025-12-04T12:26:06.7710886Z  * [new branch]                gh/anijain2305/950/orig -> origin/gh/anijain2305/950/orig
2025-12-04T12:26:06.7711083Z  * [new branch]                gh/anijain2305/951/base -> origin/gh/anijain2305/951/base
2025-12-04T12:26:06.7711269Z  * [new branch]                gh/anijain2305/951/head -> origin/gh/anijain2305/951/head
2025-12-04T12:26:06.7711458Z  * [new branch]                gh/anijain2305/951/orig -> origin/gh/anijain2305/951/orig
2025-12-04T12:26:06.7711649Z  * [new branch]                gh/anijain2305/952/base -> origin/gh/anijain2305/952/base
2025-12-04T12:26:06.7711835Z  * [new branch]                gh/anijain2305/952/head -> origin/gh/anijain2305/952/head
2025-12-04T12:26:06.7712027Z  * [new branch]                gh/anijain2305/952/orig -> origin/gh/anijain2305/952/orig
2025-12-04T12:26:06.7712222Z  * [new branch]                gh/anijain2305/953/base -> origin/gh/anijain2305/953/base
2025-12-04T12:26:06.7712409Z  * [new branch]                gh/anijain2305/953/head -> origin/gh/anijain2305/953/head
2025-12-04T12:26:06.7712597Z  * [new branch]                gh/anijain2305/953/orig -> origin/gh/anijain2305/953/orig
2025-12-04T12:26:06.7712789Z  * [new branch]                gh/anijain2305/954/base -> origin/gh/anijain2305/954/base
2025-12-04T12:26:06.7712977Z  * [new branch]                gh/anijain2305/954/head -> origin/gh/anijain2305/954/head
2025-12-04T12:26:06.7713168Z  * [new branch]                gh/anijain2305/954/orig -> origin/gh/anijain2305/954/orig
2025-12-04T12:26:06.7713359Z  * [new branch]                gh/anijain2305/955/base -> origin/gh/anijain2305/955/base
2025-12-04T12:26:06.7713546Z  * [new branch]                gh/anijain2305/955/head -> origin/gh/anijain2305/955/head
2025-12-04T12:26:06.7713735Z  * [new branch]                gh/anijain2305/955/orig -> origin/gh/anijain2305/955/orig
2025-12-04T12:26:06.7713957Z  * [new branch]                gh/anijain2305/956/base -> origin/gh/anijain2305/956/base
2025-12-04T12:26:06.7714145Z  * [new branch]                gh/anijain2305/956/head -> origin/gh/anijain2305/956/head
2025-12-04T12:26:06.7714365Z  * [new branch]                gh/anijain2305/956/orig -> origin/gh/anijain2305/956/orig
2025-12-04T12:26:06.7714555Z  * [new branch]                gh/anijain2305/957/base -> origin/gh/anijain2305/957/base
2025-12-04T12:26:06.7714743Z  * [new branch]                gh/anijain2305/957/head -> origin/gh/anijain2305/957/head
2025-12-04T12:26:06.7714931Z  * [new branch]                gh/anijain2305/957/orig -> origin/gh/anijain2305/957/orig
2025-12-04T12:26:06.7715117Z  * [new branch]                gh/anijain2305/958/base -> origin/gh/anijain2305/958/base
2025-12-04T12:26:06.7715311Z  * [new branch]                gh/anijain2305/958/head -> origin/gh/anijain2305/958/head
2025-12-04T12:26:06.7715502Z  * [new branch]                gh/anijain2305/958/orig -> origin/gh/anijain2305/958/orig
2025-12-04T12:26:06.7715691Z  * [new branch]                gh/anijain2305/959/base -> origin/gh/anijain2305/959/base
2025-12-04T12:26:06.7715880Z  * [new branch]                gh/anijain2305/959/head -> origin/gh/anijain2305/959/head
2025-12-04T12:26:06.7716076Z  * [new branch]                gh/anijain2305/959/orig -> origin/gh/anijain2305/959/orig
2025-12-04T12:26:06.7716267Z  * [new branch]                gh/anijain2305/960/base -> origin/gh/anijain2305/960/base
2025-12-04T12:26:06.7716456Z  * [new branch]                gh/anijain2305/960/head -> origin/gh/anijain2305/960/head
2025-12-04T12:26:06.7716644Z  * [new branch]                gh/anijain2305/960/orig -> origin/gh/anijain2305/960/orig
2025-12-04T12:26:06.7716863Z  * [new branch]                gh/anijain2305/961/base -> origin/gh/anijain2305/961/base
2025-12-04T12:26:06.7717053Z  * [new branch]                gh/anijain2305/961/head -> origin/gh/anijain2305/961/head
2025-12-04T12:26:06.7717248Z  * [new branch]                gh/anijain2305/961/orig -> origin/gh/anijain2305/961/orig
2025-12-04T12:26:06.7717441Z  * [new branch]                gh/anijain2305/962/base -> origin/gh/anijain2305/962/base
2025-12-04T12:26:06.7717633Z  * [new branch]                gh/anijain2305/962/head -> origin/gh/anijain2305/962/head
2025-12-04T12:26:06.7717827Z  * [new branch]                gh/anijain2305/962/orig -> origin/gh/anijain2305/962/orig
2025-12-04T12:26:06.7718013Z  * [new branch]                gh/anijain2305/963/base -> origin/gh/anijain2305/963/base
2025-12-04T12:26:06.7718205Z  * [new branch]                gh/anijain2305/963/head -> origin/gh/anijain2305/963/head
2025-12-04T12:26:06.7718398Z  * [new branch]                gh/anijain2305/963/orig -> origin/gh/anijain2305/963/orig
2025-12-04T12:26:06.7718584Z  * [new branch]                gh/anijain2305/964/base -> origin/gh/anijain2305/964/base
2025-12-04T12:26:06.7718776Z  * [new branch]                gh/anijain2305/964/head -> origin/gh/anijain2305/964/head
2025-12-04T12:26:06.7718967Z  * [new branch]                gh/anijain2305/964/orig -> origin/gh/anijain2305/964/orig
2025-12-04T12:26:06.7719157Z  * [new branch]                gh/anijain2305/965/base -> origin/gh/anijain2305/965/base
2025-12-04T12:26:06.7719355Z  * [new branch]                gh/anijain2305/965/head -> origin/gh/anijain2305/965/head
2025-12-04T12:26:06.7719545Z  * [new branch]                gh/anijain2305/965/orig -> origin/gh/anijain2305/965/orig
2025-12-04T12:26:06.7719733Z  * [new branch]                gh/anijain2305/966/base -> origin/gh/anijain2305/966/base
2025-12-04T12:26:06.7719923Z  * [new branch]                gh/anijain2305/966/head -> origin/gh/anijain2305/966/head
2025-12-04T12:26:06.7720109Z  * [new branch]                gh/anijain2305/966/orig -> origin/gh/anijain2305/966/orig
2025-12-04T12:26:06.7720303Z  * [new branch]                gh/anijain2305/967/base -> origin/gh/anijain2305/967/base
2025-12-04T12:26:06.7720492Z  * [new branch]                gh/anijain2305/967/head -> origin/gh/anijain2305/967/head
2025-12-04T12:26:06.7720734Z  * [new branch]                gh/anijain2305/967/orig -> origin/gh/anijain2305/967/orig
2025-12-04T12:26:06.7720925Z  * [new branch]                gh/anijain2305/968/base -> origin/gh/anijain2305/968/base
2025-12-04T12:26:06.7721159Z  * [new branch]                gh/anijain2305/968/head -> origin/gh/anijain2305/968/head
2025-12-04T12:26:06.7721345Z  * [new branch]                gh/anijain2305/968/orig -> origin/gh/anijain2305/968/orig
2025-12-04T12:26:06.7721538Z  * [new branch]                gh/anijain2305/969/base -> origin/gh/anijain2305/969/base
2025-12-04T12:26:06.7721728Z  * [new branch]                gh/anijain2305/969/head -> origin/gh/anijain2305/969/head
2025-12-04T12:26:06.7721914Z  * [new branch]                gh/anijain2305/969/orig -> origin/gh/anijain2305/969/orig
2025-12-04T12:26:06.7722104Z  * [new branch]                gh/anijain2305/970/base -> origin/gh/anijain2305/970/base
2025-12-04T12:26:06.7722299Z  * [new branch]                gh/anijain2305/970/head -> origin/gh/anijain2305/970/head
2025-12-04T12:26:06.7722487Z  * [new branch]                gh/anijain2305/970/orig -> origin/gh/anijain2305/970/orig
2025-12-04T12:26:06.7722675Z  * [new branch]                gh/anjali411/216/base   -> origin/gh/anjali411/216/base
2025-12-04T12:26:06.7722865Z  * [new branch]                gh/anjali411/216/head   -> origin/gh/anjali411/216/head
2025-12-04T12:26:06.7723050Z  * [new branch]                gh/anjali411/216/orig   -> origin/gh/anjali411/216/orig
2025-12-04T12:26:06.7723242Z  * [new branch]                gh/anshul-si/1/base     -> origin/gh/anshul-si/1/base
2025-12-04T12:26:06.7723430Z  * [new branch]                gh/anshul-si/1/head     -> origin/gh/anshul-si/1/head
2025-12-04T12:26:06.7723615Z  * [new branch]                gh/anshul-si/2/base     -> origin/gh/anshul-si/2/base
2025-12-04T12:26:06.7723798Z  * [new branch]                gh/anshul-si/2/head     -> origin/gh/anshul-si/2/head
2025-12-04T12:26:06.7723986Z  * [new branch]                gh/anshul-si/3/base     -> origin/gh/anshul-si/3/base
2025-12-04T12:26:06.7724168Z  * [new branch]                gh/anshul-si/3/head     -> origin/gh/anshul-si/3/head
2025-12-04T12:26:06.7724356Z  * [new branch]                gh/anshul-si/4/base     -> origin/gh/anshul-si/4/base
2025-12-04T12:26:06.7724541Z  * [new branch]                gh/anshul-si/4/head     -> origin/gh/anshul-si/4/head
2025-12-04T12:26:06.7724726Z  * [new branch]                gh/anshul-si/5/base     -> origin/gh/anshul-si/5/base
2025-12-04T12:26:06.7724911Z  * [new branch]                gh/anshul-si/5/head     -> origin/gh/anshul-si/5/head
2025-12-04T12:26:06.7725095Z  * [new branch]                gh/anshul-si/53/base    -> origin/gh/anshul-si/53/base
2025-12-04T12:26:06.7725283Z  * [new branch]                gh/anshul-si/53/head    -> origin/gh/anshul-si/53/head
2025-12-04T12:26:06.7725470Z  * [new branch]                gh/anshul-si/58/base    -> origin/gh/anshul-si/58/base
2025-12-04T12:26:06.7725654Z  * [new branch]                gh/anshul-si/58/head    -> origin/gh/anshul-si/58/head
2025-12-04T12:26:06.7725838Z  * [new branch]                gh/anshul-si/66/base    -> origin/gh/anshul-si/66/base
2025-12-04T12:26:06.7726023Z  * [new branch]                gh/anshul-si/66/head    -> origin/gh/anshul-si/66/head
2025-12-04T12:26:06.7726207Z  * [new branch]                gh/anshul-si/66/orig    -> origin/gh/anshul-si/66/orig
2025-12-04T12:26:06.7726395Z  * [new branch]                gh/anshul-si/67/base    -> origin/gh/anshul-si/67/base
2025-12-04T12:26:06.7726581Z  * [new branch]                gh/anshul-si/67/head    -> origin/gh/anshul-si/67/head
2025-12-04T12:26:06.7726805Z  * [new branch]                gh/anshul-si/67/orig    -> origin/gh/anshul-si/67/orig
2025-12-04T12:26:06.7726993Z  * [new branch]                gh/anshul-si/68/base    -> origin/gh/anshul-si/68/base
2025-12-04T12:26:06.7727178Z  * [new branch]                gh/anshul-si/68/head    -> origin/gh/anshul-si/68/head
2025-12-04T12:26:06.7727407Z  * [new branch]                gh/anshul-si/68/orig    -> origin/gh/anshul-si/68/orig
2025-12-04T12:26:06.7727593Z  * [new branch]                gh/anshul-si/69/base    -> origin/gh/anshul-si/69/base
2025-12-04T12:26:06.7727779Z  * [new branch]                gh/anshul-si/69/head    -> origin/gh/anshul-si/69/head
2025-12-04T12:26:06.7727995Z  * [new branch]                gh/anshul-si/69/orig    -> origin/gh/anshul-si/69/orig
2025-12-04T12:26:06.7728186Z  * [new branch]                gh/anshul-si/70/base    -> origin/gh/anshul-si/70/base
2025-12-04T12:26:06.7728366Z  * [new branch]                gh/anshul-si/70/head    -> origin/gh/anshul-si/70/head
2025-12-04T12:26:06.7728550Z  * [new branch]                gh/anshul-si/70/orig    -> origin/gh/anshul-si/70/orig
2025-12-04T12:26:06.7728734Z  * [new branch]                gh/anshul-si/71/base    -> origin/gh/anshul-si/71/base
2025-12-04T12:26:06.7728914Z  * [new branch]                gh/anshul-si/71/head    -> origin/gh/anshul-si/71/head
2025-12-04T12:26:06.7729104Z  * [new branch]                gh/anshul-si/71/orig    -> origin/gh/anshul-si/71/orig
2025-12-04T12:26:06.7729292Z  * [new branch]                gh/anshul-si/72/base    -> origin/gh/anshul-si/72/base
2025-12-04T12:26:06.7729476Z  * [new branch]                gh/anshul-si/72/head    -> origin/gh/anshul-si/72/head
2025-12-04T12:26:06.7729658Z  * [new branch]                gh/anshul-si/72/orig    -> origin/gh/anshul-si/72/orig
2025-12-04T12:26:06.7729841Z  * [new branch]                gh/anshul-si/73/base    -> origin/gh/anshul-si/73/base
2025-12-04T12:26:06.7730025Z  * [new branch]                gh/anshul-si/73/head    -> origin/gh/anshul-si/73/head
2025-12-04T12:26:06.7730206Z  * [new branch]                gh/anshul-si/73/orig    -> origin/gh/anshul-si/73/orig
2025-12-04T12:26:06.7730391Z  * [new branch]                gh/aorenste/132/base    -> origin/gh/aorenste/132/base
2025-12-04T12:26:06.7730575Z  * [new branch]                gh/aorenste/132/head    -> origin/gh/aorenste/132/head
2025-12-04T12:26:06.7730764Z  * [new branch]                gh/aorenste/134/base    -> origin/gh/aorenste/134/base
2025-12-04T12:26:06.7730951Z  * [new branch]                gh/aorenste/134/head    -> origin/gh/aorenste/134/head
2025-12-04T12:26:06.7731138Z  * [new branch]                gh/aorenste/134/orig    -> origin/gh/aorenste/134/orig
2025-12-04T12:26:06.7731325Z  * [new branch]                gh/aorenste/139/base    -> origin/gh/aorenste/139/base
2025-12-04T12:26:06.7731511Z  * [new branch]                gh/aorenste/139/head    -> origin/gh/aorenste/139/head
2025-12-04T12:26:06.7731693Z  * [new branch]                gh/aorenste/139/orig    -> origin/gh/aorenste/139/orig
2025-12-04T12:26:06.7731882Z  * [new branch]                gh/aorenste/141/base    -> origin/gh/aorenste/141/base
2025-12-04T12:26:06.7732069Z  * [new branch]                gh/aorenste/141/head    -> origin/gh/aorenste/141/head
2025-12-04T12:26:06.7732252Z  * [new branch]                gh/aorenste/145/base    -> origin/gh/aorenste/145/base
2025-12-04T12:26:06.7732438Z  * [new branch]                gh/aorenste/145/head    -> origin/gh/aorenste/145/head
2025-12-04T12:26:06.7732622Z  * [new branch]                gh/aorenste/145/orig    -> origin/gh/aorenste/145/orig
2025-12-04T12:26:06.7732816Z  * [new branch]                gh/aorenste/146/base    -> origin/gh/aorenste/146/base
2025-12-04T12:26:06.7733003Z  * [new branch]                gh/aorenste/146/head    -> origin/gh/aorenste/146/head
2025-12-04T12:26:06.7733185Z  * [new branch]                gh/aorenste/146/orig    -> origin/gh/aorenste/146/orig
2025-12-04T12:26:06.7733372Z  * [new branch]                gh/aorenste/147/base    -> origin/gh/aorenste/147/base
2025-12-04T12:26:06.7733558Z  * [new branch]                gh/aorenste/147/head    -> origin/gh/aorenste/147/head
2025-12-04T12:26:06.7733743Z  * [new branch]                gh/aorenste/147/orig    -> origin/gh/aorenste/147/orig
2025-12-04T12:26:06.7733931Z  * [new branch]                gh/aorenste/148/base    -> origin/gh/aorenste/148/base
2025-12-04T12:26:06.7734157Z  * [new branch]                gh/aorenste/148/head    -> origin/gh/aorenste/148/head
2025-12-04T12:26:06.7734342Z  * [new branch]                gh/aorenste/148/orig    -> origin/gh/aorenste/148/orig
2025-12-04T12:26:06.7734564Z  * [new branch]                gh/aorenste/149/base    -> origin/gh/aorenste/149/base
2025-12-04T12:26:06.7734750Z  * [new branch]                gh/aorenste/149/head    -> origin/gh/aorenste/149/head
2025-12-04T12:26:06.7734932Z  * [new branch]                gh/aorenste/149/orig    -> origin/gh/aorenste/149/orig
2025-12-04T12:26:06.7735120Z  * [new branch]                gh/aorenste/150/base    -> origin/gh/aorenste/150/base
2025-12-04T12:26:06.7735309Z  * [new branch]                gh/aorenste/150/head    -> origin/gh/aorenste/150/head
2025-12-04T12:26:06.7735492Z  * [new branch]                gh/aorenste/150/orig    -> origin/gh/aorenste/150/orig
2025-12-04T12:26:06.7735677Z  * [new branch]                gh/aorenste/151/base    -> origin/gh/aorenste/151/base
2025-12-04T12:26:06.7735869Z  * [new branch]                gh/aorenste/151/head    -> origin/gh/aorenste/151/head
2025-12-04T12:26:06.7736052Z  * [new branch]                gh/aorenste/151/orig    -> origin/gh/aorenste/151/orig
2025-12-04T12:26:06.7736242Z  * [new branch]                gh/aorenste/152/base    -> origin/gh/aorenste/152/base
2025-12-04T12:26:06.7736428Z  * [new branch]                gh/aorenste/152/head    -> origin/gh/aorenste/152/head
2025-12-04T12:26:06.7736613Z  * [new branch]                gh/aorenste/152/orig    -> origin/gh/aorenste/152/orig
2025-12-04T12:26:06.7736857Z  * [new branch]                gh/aorenste/153/base    -> origin/gh/aorenste/153/base
2025-12-04T12:26:06.7737044Z  * [new branch]                gh/aorenste/153/head    -> origin/gh/aorenste/153/head
2025-12-04T12:26:06.7737234Z  * [new branch]                gh/aorenste/153/orig    -> origin/gh/aorenste/153/orig
2025-12-04T12:26:06.7737424Z  * [new branch]                gh/aorenste/154/base    -> origin/gh/aorenste/154/base
2025-12-04T12:26:06.7737607Z  * [new branch]                gh/aorenste/154/head    -> origin/gh/aorenste/154/head
2025-12-04T12:26:06.7737792Z  * [new branch]                gh/aorenste/154/orig    -> origin/gh/aorenste/154/orig
2025-12-04T12:26:06.7737985Z  * [new branch]                gh/aorenste/155/base    -> origin/gh/aorenste/155/base
2025-12-04T12:26:06.7738171Z  * [new branch]                gh/aorenste/155/head    -> origin/gh/aorenste/155/head
2025-12-04T12:26:06.7738358Z  * [new branch]                gh/aorenste/155/orig    -> origin/gh/aorenste/155/orig
2025-12-04T12:26:06.7738545Z  * [new branch]                gh/aorenste/156/base    -> origin/gh/aorenste/156/base
2025-12-04T12:26:06.7738728Z  * [new branch]                gh/aorenste/156/head    -> origin/gh/aorenste/156/head
2025-12-04T12:26:06.7738915Z  * [new branch]                gh/aorenste/156/orig    -> origin/gh/aorenste/156/orig
2025-12-04T12:26:06.7739106Z  * [new branch]                gh/aorenste/157/base    -> origin/gh/aorenste/157/base
2025-12-04T12:26:06.7739287Z  * [new branch]                gh/aorenste/157/head    -> origin/gh/aorenste/157/head
2025-12-04T12:26:06.7739470Z  * [new branch]                gh/aorenste/157/orig    -> origin/gh/aorenste/157/orig
2025-12-04T12:26:06.7739656Z  * [new branch]                gh/aorenste/158/base    -> origin/gh/aorenste/158/base
2025-12-04T12:26:06.7739840Z  * [new branch]                gh/aorenste/158/head    -> origin/gh/aorenste/158/head
2025-12-04T12:26:06.7740031Z  * [new branch]                gh/aorenste/158/orig    -> origin/gh/aorenste/158/orig
2025-12-04T12:26:06.7740218Z  * [new branch]                gh/aorenste/159/base    -> origin/gh/aorenste/159/base
2025-12-04T12:26:06.7740403Z  * [new branch]                gh/aorenste/159/head    -> origin/gh/aorenste/159/head
2025-12-04T12:26:06.7740591Z  * [new branch]                gh/aorenste/159/orig    -> origin/gh/aorenste/159/orig
2025-12-04T12:26:06.7740827Z  * [new branch]                gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base
2025-12-04T12:26:06.7741030Z  * [new branch]                gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head
2025-12-04T12:26:06.7741230Z  * [new branch]                gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base
2025-12-04T12:26:06.7741469Z  * [new branch]                gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head
2025-12-04T12:26:06.7741669Z  * [new branch]                gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig
2025-12-04T12:26:06.7741862Z  * [new branch]                gh/bdhirsh/666/base     -> origin/gh/bdhirsh/666/base
2025-12-04T12:26:06.7742044Z  * [new branch]                gh/bdhirsh/666/head     -> origin/gh/bdhirsh/666/head
2025-12-04T12:26:06.7742230Z  * [new branch]                gh/bdhirsh/666/orig     -> origin/gh/bdhirsh/666/orig
2025-12-04T12:26:06.7742416Z  * [new branch]                gh/bdhirsh/668/base     -> origin/gh/bdhirsh/668/base
2025-12-04T12:26:06.7742603Z  * [new branch]                gh/bdhirsh/668/head     -> origin/gh/bdhirsh/668/head
2025-12-04T12:26:06.7742789Z  * [new branch]                gh/bdhirsh/668/orig     -> origin/gh/bdhirsh/668/orig
2025-12-04T12:26:06.7742971Z  * [new branch]                gh/bdhirsh/669/base     -> origin/gh/bdhirsh/669/base
2025-12-04T12:26:06.7743153Z  * [new branch]                gh/bdhirsh/669/head     -> origin/gh/bdhirsh/669/head
2025-12-04T12:26:06.7743339Z  * [new branch]                gh/bdhirsh/669/orig     -> origin/gh/bdhirsh/669/orig
2025-12-04T12:26:06.7743521Z  * [new branch]                gh/bdhirsh/670/base     -> origin/gh/bdhirsh/670/base
2025-12-04T12:26:06.7743701Z  * [new branch]                gh/bdhirsh/670/head     -> origin/gh/bdhirsh/670/head
2025-12-04T12:26:06.7743885Z  * [new branch]                gh/bdhirsh/670/orig     -> origin/gh/bdhirsh/670/orig
2025-12-04T12:26:06.7744070Z  * [new branch]                gh/bdhirsh/672/base     -> origin/gh/bdhirsh/672/base
2025-12-04T12:26:06.7744251Z  * [new branch]                gh/bdhirsh/672/head     -> origin/gh/bdhirsh/672/head
2025-12-04T12:26:06.7744434Z  * [new branch]                gh/bdhirsh/672/orig     -> origin/gh/bdhirsh/672/orig
2025-12-04T12:26:06.7744617Z  * [new branch]                gh/bdhirsh/675/base     -> origin/gh/bdhirsh/675/base
2025-12-04T12:26:06.7744802Z  * [new branch]                gh/bdhirsh/675/head     -> origin/gh/bdhirsh/675/head
2025-12-04T12:26:06.7744984Z  * [new branch]                gh/bdhirsh/675/orig     -> origin/gh/bdhirsh/675/orig
2025-12-04T12:26:06.7745162Z  * [new branch]                gh/bdhirsh/676/base     -> origin/gh/bdhirsh/676/base
2025-12-04T12:26:06.7745345Z  * [new branch]                gh/bdhirsh/676/head     -> origin/gh/bdhirsh/676/head
2025-12-04T12:26:06.7745529Z  * [new branch]                gh/bdhirsh/676/orig     -> origin/gh/bdhirsh/676/orig
2025-12-04T12:26:06.7745602Z  * [new branch]                gh/bdhirsh/677/base     -> origin/gh/bdhirsh/677/base
2025-12-04T12:26:06.7745676Z  * [new branch]                gh/bdhirsh/677/head     -> origin/gh/bdhirsh/677/head
2025-12-04T12:26:06.7745749Z  * [new branch]                gh/bdhirsh/677/orig     -> origin/gh/bdhirsh/677/orig
2025-12-04T12:26:06.7745820Z  * [new branch]                gh/bdhirsh/678/base     -> origin/gh/bdhirsh/678/base
2025-12-04T12:26:06.7745895Z  * [new branch]                gh/bdhirsh/678/head     -> origin/gh/bdhirsh/678/head
2025-12-04T12:26:06.7745965Z  * [new branch]                gh/bdhirsh/678/orig     -> origin/gh/bdhirsh/678/orig
2025-12-04T12:26:06.7746037Z  * [new branch]                gh/bdhirsh/679/base     -> origin/gh/bdhirsh/679/base
2025-12-04T12:26:06.7746110Z  * [new branch]                gh/bdhirsh/679/head     -> origin/gh/bdhirsh/679/head
2025-12-04T12:26:06.7746181Z  * [new branch]                gh/bdhirsh/679/orig     -> origin/gh/bdhirsh/679/orig
2025-12-04T12:26:06.7746252Z  * [new branch]                gh/bdhirsh/680/base     -> origin/gh/bdhirsh/680/base
2025-12-04T12:26:06.7746370Z  * [new branch]                gh/bdhirsh/680/head     -> origin/gh/bdhirsh/680/head
2025-12-04T12:26:06.7746442Z  * [new branch]                gh/bdhirsh/680/orig     -> origin/gh/bdhirsh/680/orig
2025-12-04T12:26:06.7746512Z  * [new branch]                gh/bdhirsh/681/base     -> origin/gh/bdhirsh/681/base
2025-12-04T12:26:06.7746612Z  * [new branch]                gh/bdhirsh/681/head     -> origin/gh/bdhirsh/681/head
2025-12-04T12:26:06.7746683Z  * [new branch]                gh/bdhirsh/681/orig     -> origin/gh/bdhirsh/681/orig
2025-12-04T12:26:06.7746809Z  * [new branch]                gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base
2025-12-04T12:26:06.7746904Z  * [new branch]                gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head
2025-12-04T12:26:06.7746992Z  * [new branch]                gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig
2025-12-04T12:26:06.7747080Z  * [new branch]                gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base
2025-12-04T12:26:06.7747173Z  * [new branch]                gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head
2025-12-04T12:26:06.7747260Z  * [new branch]                gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig
2025-12-04T12:26:06.7747351Z  * [new branch]                gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base
2025-12-04T12:26:06.7747443Z  * [new branch]                gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head
2025-12-04T12:26:06.7747529Z  * [new branch]                gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig
2025-12-04T12:26:06.7747618Z  * [new branch]                gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base
2025-12-04T12:26:06.7747705Z  * [new branch]                gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head
2025-12-04T12:26:06.7747792Z  * [new branch]                gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig
2025-12-04T12:26:06.7747883Z  * [new branch]                gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base
2025-12-04T12:26:06.7747970Z  * [new branch]                gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head
2025-12-04T12:26:06.7748060Z  * [new branch]                gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig
2025-12-04T12:26:06.7748151Z  * [new branch]                gh/benjaminglass1/109/base -> origin/gh/benjaminglass1/109/base
2025-12-04T12:26:06.7748239Z  * [new branch]                gh/benjaminglass1/109/head -> origin/gh/benjaminglass1/109/head
2025-12-04T12:26:06.7748326Z  * [new branch]                gh/benjaminglass1/109/orig -> origin/gh/benjaminglass1/109/orig
2025-12-04T12:26:06.7748416Z  * [new branch]                gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base
2025-12-04T12:26:06.7748500Z  * [new branch]                gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head
2025-12-04T12:26:06.7748585Z  * [new branch]                gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig
2025-12-04T12:26:06.7748667Z  * [new branch]                gh/bobrenjc93/570/base  -> origin/gh/bobrenjc93/570/base
2025-12-04T12:26:06.7748745Z  * [new branch]                gh/bobrenjc93/570/head  -> origin/gh/bobrenjc93/570/head
2025-12-04T12:26:06.7748822Z  * [new branch]                gh/bobrenjc93/570/orig  -> origin/gh/bobrenjc93/570/orig
2025-12-04T12:26:06.7748901Z  * [new branch]                gh/bobrenjc93/604/base  -> origin/gh/bobrenjc93/604/base
2025-12-04T12:26:06.7748977Z  * [new branch]                gh/bobrenjc93/604/head  -> origin/gh/bobrenjc93/604/head
2025-12-04T12:26:06.7749051Z  * [new branch]                gh/bobrenjc93/604/orig  -> origin/gh/bobrenjc93/604/orig
2025-12-04T12:26:06.7749129Z  * [new branch]                gh/bobrenjc93/638/base  -> origin/gh/bobrenjc93/638/base
2025-12-04T12:26:06.7749206Z  * [new branch]                gh/bobrenjc93/638/head  -> origin/gh/bobrenjc93/638/head
2025-12-04T12:26:06.7749322Z  * [new branch]                gh/bobrenjc93/638/orig  -> origin/gh/bobrenjc93/638/orig
2025-12-04T12:26:06.7749397Z  * [new branch]                gh/bobrenjc93/653/base  -> origin/gh/bobrenjc93/653/base
2025-12-04T12:26:06.7749516Z  * [new branch]                gh/bobrenjc93/653/head  -> origin/gh/bobrenjc93/653/head
2025-12-04T12:26:06.7749594Z  * [new branch]                gh/bobrenjc93/653/orig  -> origin/gh/bobrenjc93/653/orig
2025-12-04T12:26:06.7749668Z  * [new branch]                gh/bobrenjc93/654/base  -> origin/gh/bobrenjc93/654/base
2025-12-04T12:26:06.7749743Z  * [new branch]                gh/bobrenjc93/654/head  -> origin/gh/bobrenjc93/654/head
2025-12-04T12:26:06.7749819Z  * [new branch]                gh/bobrenjc93/654/orig  -> origin/gh/bobrenjc93/654/orig
2025-12-04T12:26:06.7749893Z  * [new branch]                gh/bobrenjc93/657/base  -> origin/gh/bobrenjc93/657/base
2025-12-04T12:26:06.7749966Z  * [new branch]                gh/bobrenjc93/657/head  -> origin/gh/bobrenjc93/657/head
2025-12-04T12:26:06.7750046Z  * [new branch]                gh/bobrenjc93/657/orig  -> origin/gh/bobrenjc93/657/orig
2025-12-04T12:26:06.7750124Z  * [new branch]                gh/bobrenjc93/672/base  -> origin/gh/bobrenjc93/672/base
2025-12-04T12:26:06.7750198Z  * [new branch]                gh/bobrenjc93/672/head  -> origin/gh/bobrenjc93/672/head
2025-12-04T12:26:06.7750277Z  * [new branch]                gh/bobrenjc93/672/orig  -> origin/gh/bobrenjc93/672/orig
2025-12-04T12:26:06.7750350Z  * [new branch]                gh/bobrenjc93/679/base  -> origin/gh/bobrenjc93/679/base
2025-12-04T12:26:06.7750425Z  * [new branch]                gh/bobrenjc93/679/head  -> origin/gh/bobrenjc93/679/head
2025-12-04T12:26:06.7750503Z  * [new branch]                gh/bobrenjc93/679/orig  -> origin/gh/bobrenjc93/679/orig
2025-12-04T12:26:06.7750576Z  * [new branch]                gh/bobrenjc93/680/base  -> origin/gh/bobrenjc93/680/base
2025-12-04T12:26:06.7750652Z  * [new branch]                gh/bobrenjc93/680/head  -> origin/gh/bobrenjc93/680/head
2025-12-04T12:26:06.7750731Z  * [new branch]                gh/bobrenjc93/680/orig  -> origin/gh/bobrenjc93/680/orig
2025-12-04T12:26:06.7750804Z  * [new branch]                gh/bobrenjc93/681/base  -> origin/gh/bobrenjc93/681/base
2025-12-04T12:26:06.7750886Z  * [new branch]                gh/bobrenjc93/681/head  -> origin/gh/bobrenjc93/681/head
2025-12-04T12:26:06.7750961Z  * [new branch]                gh/bobrenjc93/681/orig  -> origin/gh/bobrenjc93/681/orig
2025-12-04T12:26:06.7751035Z  * [new branch]                gh/bobrenjc93/682/base  -> origin/gh/bobrenjc93/682/base
2025-12-04T12:26:06.7751112Z  * [new branch]                gh/bobrenjc93/682/head  -> origin/gh/bobrenjc93/682/head
2025-12-04T12:26:06.7751185Z  * [new branch]                gh/bobrenjc93/682/orig  -> origin/gh/bobrenjc93/682/orig
2025-12-04T12:26:06.7751257Z  * [new branch]                gh/bobrenjc93/683/base  -> origin/gh/bobrenjc93/683/base
2025-12-04T12:26:06.7751337Z  * [new branch]                gh/bobrenjc93/683/head  -> origin/gh/bobrenjc93/683/head
2025-12-04T12:26:06.7751411Z  * [new branch]                gh/bobrenjc93/683/orig  -> origin/gh/bobrenjc93/683/orig
2025-12-04T12:26:06.7751485Z  * [new branch]                gh/bobrenjc93/684/base  -> origin/gh/bobrenjc93/684/base
2025-12-04T12:26:06.7751562Z  * [new branch]                gh/bobrenjc93/684/head  -> origin/gh/bobrenjc93/684/head
2025-12-04T12:26:06.7751634Z  * [new branch]                gh/bobrenjc93/684/orig  -> origin/gh/bobrenjc93/684/orig
2025-12-04T12:26:06.7751707Z  * [new branch]                gh/bobrenjc93/685/base  -> origin/gh/bobrenjc93/685/base
2025-12-04T12:26:06.7751782Z  * [new branch]                gh/bobrenjc93/685/head  -> origin/gh/bobrenjc93/685/head
2025-12-04T12:26:06.7751854Z  * [new branch]                gh/bobrenjc93/685/orig  -> origin/gh/bobrenjc93/685/orig
2025-12-04T12:26:06.7751927Z  * [new branch]                gh/bobrenjc93/686/base  -> origin/gh/bobrenjc93/686/base
2025-12-04T12:26:06.7752032Z  * [new branch]                gh/bobrenjc93/686/head  -> origin/gh/bobrenjc93/686/head
2025-12-04T12:26:06.7752105Z  * [new branch]                gh/bobrenjc93/686/orig  -> origin/gh/bobrenjc93/686/orig
2025-12-04T12:26:06.7752201Z  * [new branch]                gh/bobrenjc93/687/base  -> origin/gh/bobrenjc93/687/base
2025-12-04T12:26:06.7752278Z  * [new branch]                gh/bobrenjc93/687/head  -> origin/gh/bobrenjc93/687/head
2025-12-04T12:26:06.7752351Z  * [new branch]                gh/bobrenjc93/687/orig  -> origin/gh/bobrenjc93/687/orig
2025-12-04T12:26:06.7752424Z  * [new branch]                gh/bobrenjc93/688/base  -> origin/gh/bobrenjc93/688/base
2025-12-04T12:26:06.7752500Z  * [new branch]                gh/bobrenjc93/688/head  -> origin/gh/bobrenjc93/688/head
2025-12-04T12:26:06.7752573Z  * [new branch]                gh/bobrenjc93/688/orig  -> origin/gh/bobrenjc93/688/orig
2025-12-04T12:26:06.7752648Z  * [new branch]                gh/bobrenjc93/689/base  -> origin/gh/bobrenjc93/689/base
2025-12-04T12:26:06.7752722Z  * [new branch]                gh/bobrenjc93/689/head  -> origin/gh/bobrenjc93/689/head
2025-12-04T12:26:06.7752796Z  * [new branch]                gh/bobrenjc93/689/orig  -> origin/gh/bobrenjc93/689/orig
2025-12-04T12:26:06.7752874Z  * [new branch]                gh/bobrenjc93/690/base  -> origin/gh/bobrenjc93/690/base
2025-12-04T12:26:06.7752949Z  * [new branch]                gh/bobrenjc93/690/head  -> origin/gh/bobrenjc93/690/head
2025-12-04T12:26:06.7753022Z  * [new branch]                gh/bobrenjc93/690/orig  -> origin/gh/bobrenjc93/690/orig
2025-12-04T12:26:06.7753097Z  * [new branch]                gh/bobrenjc93/691/base  -> origin/gh/bobrenjc93/691/base
2025-12-04T12:26:06.7753170Z  * [new branch]                gh/bobrenjc93/691/head  -> origin/gh/bobrenjc93/691/head
2025-12-04T12:26:06.7753242Z  * [new branch]                gh/bobrenjc93/691/orig  -> origin/gh/bobrenjc93/691/orig
2025-12-04T12:26:06.7753320Z  * [new branch]                gh/bobrenjc93/692/base  -> origin/gh/bobrenjc93/692/base
2025-12-04T12:26:06.7753393Z  * [new branch]                gh/bobrenjc93/692/head  -> origin/gh/bobrenjc93/692/head
2025-12-04T12:26:06.7753466Z  * [new branch]                gh/bobrenjc93/692/orig  -> origin/gh/bobrenjc93/692/orig
2025-12-04T12:26:06.7753544Z  * [new branch]                gh/bobrenjc93/693/base  -> origin/gh/bobrenjc93/693/base
2025-12-04T12:26:06.7753618Z  * [new branch]                gh/bobrenjc93/693/head  -> origin/gh/bobrenjc93/693/head
2025-12-04T12:26:06.7753693Z  * [new branch]                gh/bobrenjc93/693/orig  -> origin/gh/bobrenjc93/693/orig
2025-12-04T12:26:06.7753769Z  * [new branch]                gh/bobrenjc93/694/base  -> origin/gh/bobrenjc93/694/base
2025-12-04T12:26:06.7753842Z  * [new branch]                gh/bobrenjc93/694/head  -> origin/gh/bobrenjc93/694/head
2025-12-04T12:26:06.7753915Z  * [new branch]                gh/bobrenjc93/694/orig  -> origin/gh/bobrenjc93/694/orig
2025-12-04T12:26:06.7753990Z  * [new branch]                gh/bobrenjc93/695/base  -> origin/gh/bobrenjc93/695/base
2025-12-04T12:26:06.7754063Z  * [new branch]                gh/bobrenjc93/695/head  -> origin/gh/bobrenjc93/695/head
2025-12-04T12:26:06.7754139Z  * [new branch]                gh/bobrenjc93/695/orig  -> origin/gh/bobrenjc93/695/orig
2025-12-04T12:26:06.7754209Z  * [new branch]                gh/c00w/23/base         -> origin/gh/c00w/23/base
2025-12-04T12:26:06.7754276Z  * [new branch]                gh/c00w/23/head         -> origin/gh/c00w/23/head
2025-12-04T12:26:06.7754345Z  * [new branch]                gh/c00w/53/base         -> origin/gh/c00w/53/base
2025-12-04T12:26:06.7754409Z  * [new branch]                gh/c00w/53/head         -> origin/gh/c00w/53/head
2025-12-04T12:26:06.7754472Z  * [new branch]                gh/c00w/53/orig         -> origin/gh/c00w/53/orig
2025-12-04T12:26:06.7754538Z  * [new branch]                gh/c00w/54/base         -> origin/gh/c00w/54/base
2025-12-04T12:26:06.7754628Z  * [new branch]                gh/c00w/54/head         -> origin/gh/c00w/54/head
2025-12-04T12:26:06.7754691Z  * [new branch]                gh/c00w/54/orig         -> origin/gh/c00w/54/orig
2025-12-04T12:26:06.7754756Z  * [new branch]                gh/c00w/56/base         -> origin/gh/c00w/56/base
2025-12-04T12:26:06.7754856Z  * [new branch]                gh/c00w/56/head         -> origin/gh/c00w/56/head
2025-12-04T12:26:06.7754920Z  * [new branch]                gh/c00w/56/orig         -> origin/gh/c00w/56/orig
2025-12-04T12:26:06.7754987Z  * [new branch]                gh/c00w/57/base         -> origin/gh/c00w/57/base
2025-12-04T12:26:06.7755049Z  * [new branch]                gh/c00w/57/head         -> origin/gh/c00w/57/head
2025-12-04T12:26:06.7755111Z  * [new branch]                gh/c00w/57/orig         -> origin/gh/c00w/57/orig
2025-12-04T12:26:06.7755175Z  * [new branch]                gh/c00w/58/base         -> origin/gh/c00w/58/base
2025-12-04T12:26:06.7755238Z  * [new branch]                gh/c00w/58/head         -> origin/gh/c00w/58/head
2025-12-04T12:26:06.7755301Z  * [new branch]                gh/c00w/58/orig         -> origin/gh/c00w/58/orig
2025-12-04T12:26:06.7755375Z  * [new branch]                gh/clee2000/1/base      -> origin/gh/clee2000/1/base
2025-12-04T12:26:06.7755447Z  * [new branch]                gh/clee2000/1/head      -> origin/gh/clee2000/1/head
2025-12-04T12:26:06.7755516Z  * [new branch]                gh/clee2000/1/orig      -> origin/gh/clee2000/1/orig
2025-12-04T12:26:06.7755598Z  * [new branch]                gh/coconutruben/1/base  -> origin/gh/coconutruben/1/base
2025-12-04T12:26:06.7755678Z  * [new branch]                gh/coconutruben/1/head  -> origin/gh/coconutruben/1/head
2025-12-04T12:26:06.7755757Z  * [new branch]                gh/coconutruben/55/base -> origin/gh/coconutruben/55/base
2025-12-04T12:26:06.7755837Z  * [new branch]                gh/coconutruben/55/head -> origin/gh/coconutruben/55/head
2025-12-04T12:26:06.7755914Z  * [new branch]                gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig
2025-12-04T12:26:06.7755996Z  * [new branch]                gh/coconutruben/57/base -> origin/gh/coconutruben/57/base
2025-12-04T12:26:06.7756072Z  * [new branch]                gh/coconutruben/57/head -> origin/gh/coconutruben/57/head
2025-12-04T12:26:06.7756150Z  * [new branch]                gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig
2025-12-04T12:26:06.7756227Z  * [new branch]                gh/coconutruben/70/base -> origin/gh/coconutruben/70/base
2025-12-04T12:26:06.7756304Z  * [new branch]                gh/coconutruben/70/head -> origin/gh/coconutruben/70/head
2025-12-04T12:26:06.7756380Z  * [new branch]                gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig
2025-12-04T12:26:06.7756460Z  * [new branch]                gh/coconutruben/71/base -> origin/gh/coconutruben/71/base
2025-12-04T12:26:06.7756535Z  * [new branch]                gh/coconutruben/71/head -> origin/gh/coconutruben/71/head
2025-12-04T12:26:06.7756612Z  * [new branch]                gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig
2025-12-04T12:26:06.7756690Z  * [new branch]                gh/coconutruben/72/base -> origin/gh/coconutruben/72/base
2025-12-04T12:26:06.7756816Z  * [new branch]                gh/coconutruben/72/head -> origin/gh/coconutruben/72/head
2025-12-04T12:26:06.7756895Z  * [new branch]                gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig
2025-12-04T12:26:06.7756972Z  * [new branch]                gh/coconutruben/73/base -> origin/gh/coconutruben/73/base
2025-12-04T12:26:06.7757046Z  * [new branch]                gh/coconutruben/73/head -> origin/gh/coconutruben/73/head
2025-12-04T12:26:06.7757121Z  * [new branch]                gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig
2025-12-04T12:26:06.7757196Z  * [new branch]                gh/coconutruben/74/base -> origin/gh/coconutruben/74/base
2025-12-04T12:26:06.7757271Z  * [new branch]                gh/coconutruben/74/head -> origin/gh/coconutruben/74/head
2025-12-04T12:26:06.7757382Z  * [new branch]                gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig
2025-12-04T12:26:06.7757459Z  * [new branch]                gh/coconutruben/79/base -> origin/gh/coconutruben/79/base
2025-12-04T12:26:06.7757573Z  * [new branch]                gh/coconutruben/79/head -> origin/gh/coconutruben/79/head
2025-12-04T12:26:06.7757649Z  * [new branch]                gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig
2025-12-04T12:26:06.7757724Z  * [new branch]                gh/coconutruben/80/base -> origin/gh/coconutruben/80/base
2025-12-04T12:26:06.7757798Z  * [new branch]                gh/coconutruben/80/head -> origin/gh/coconutruben/80/head
2025-12-04T12:26:06.7757875Z  * [new branch]                gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig
2025-12-04T12:26:06.7757951Z  * [new branch]                gh/coconutruben/82/base -> origin/gh/coconutruben/82/base
2025-12-04T12:26:06.7758026Z  * [new branch]                gh/coconutruben/82/head -> origin/gh/coconutruben/82/head
2025-12-04T12:26:06.7758103Z  * [new branch]                gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig
2025-12-04T12:26:06.7758179Z  * [new branch]                gh/coconutruben/83/base -> origin/gh/coconutruben/83/base
2025-12-04T12:26:06.7758255Z  * [new branch]                gh/coconutruben/83/head -> origin/gh/coconutruben/83/head
2025-12-04T12:26:06.7758330Z  * [new branch]                gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig
2025-12-04T12:26:06.7758405Z  * [new branch]                gh/coconutruben/84/base -> origin/gh/coconutruben/84/base
2025-12-04T12:26:06.7758480Z  * [new branch]                gh/coconutruben/84/head -> origin/gh/coconutruben/84/head
2025-12-04T12:26:06.7758556Z  * [new branch]                gh/coconutruben/84/orig -> origin/gh/coconutruben/84/orig
2025-12-04T12:26:06.7758631Z  * [new branch]                gh/coconutruben/85/base -> origin/gh/coconutruben/85/base
2025-12-04T12:26:06.7758711Z  * [new branch]                gh/coconutruben/85/head -> origin/gh/coconutruben/85/head
2025-12-04T12:26:06.7758787Z  * [new branch]                gh/coconutruben/85/orig -> origin/gh/coconutruben/85/orig
2025-12-04T12:26:06.7758862Z  * [new branch]                gh/coconutruben/86/base -> origin/gh/coconutruben/86/base
2025-12-04T12:26:06.7758939Z  * [new branch]                gh/coconutruben/86/head -> origin/gh/coconutruben/86/head
2025-12-04T12:26:06.7759014Z  * [new branch]                gh/coconutruben/86/orig -> origin/gh/coconutruben/86/orig
2025-12-04T12:26:06.7759090Z  * [new branch]                gh/colinchan15/1/base   -> origin/gh/colinchan15/1/base
2025-12-04T12:26:06.7759165Z  * [new branch]                gh/colinchan15/1/head   -> origin/gh/colinchan15/1/head
2025-12-04T12:26:06.7759239Z  * [new branch]                gh/colinchan15/2/base   -> origin/gh/colinchan15/2/base
2025-12-04T12:26:06.7759312Z  * [new branch]                gh/colinchan15/2/head   -> origin/gh/colinchan15/2/head
2025-12-04T12:26:06.7759388Z  * [new branch]                gh/colinchan15/3/base   -> origin/gh/colinchan15/3/base
2025-12-04T12:26:06.7759460Z  * [new branch]                gh/colinchan15/3/head   -> origin/gh/colinchan15/3/head
2025-12-04T12:26:06.7759533Z  * [new branch]                gh/colinchan15/6/base   -> origin/gh/colinchan15/6/base
2025-12-04T12:26:06.7759609Z  * [new branch]                gh/colinchan15/6/head   -> origin/gh/colinchan15/6/head
2025-12-04T12:26:06.7759675Z  * [new branch]                gh/d4l3k/1/base         -> origin/gh/d4l3k/1/base
2025-12-04T12:26:06.7759740Z  * [new branch]                gh/d4l3k/1/head         -> origin/gh/d4l3k/1/head
2025-12-04T12:26:06.7759805Z  * [new branch]                gh/d4l3k/2/base         -> origin/gh/d4l3k/2/base
2025-12-04T12:26:06.7759869Z  * [new branch]                gh/d4l3k/2/head         -> origin/gh/d4l3k/2/head
2025-12-04T12:26:06.7759934Z  * [new branch]                gh/d4l3k/2/orig         -> origin/gh/d4l3k/2/orig
2025-12-04T12:26:06.7760027Z  * [new branch]                gh/d4l3k/3/base         -> origin/gh/d4l3k/3/base
2025-12-04T12:26:06.7760092Z  * [new branch]                gh/d4l3k/3/head         -> origin/gh/d4l3k/3/head
2025-12-04T12:26:06.7760155Z  * [new branch]                gh/d4l3k/3/orig         -> origin/gh/d4l3k/3/orig
2025-12-04T12:26:06.7760244Z  * [new branch]                gh/d4l3k/4/base         -> origin/gh/d4l3k/4/base
2025-12-04T12:26:06.7760307Z  * [new branch]                gh/d4l3k/4/head         -> origin/gh/d4l3k/4/head
2025-12-04T12:26:06.7760369Z  * [new branch]                gh/d4l3k/4/orig         -> origin/gh/d4l3k/4/orig
2025-12-04T12:26:06.7760432Z  * [new branch]                gh/d4l3k/5/base         -> origin/gh/d4l3k/5/base
2025-12-04T12:26:06.7760494Z  * [new branch]                gh/d4l3k/5/orig         -> origin/gh/d4l3k/5/orig
2025-12-04T12:26:06.7760580Z  * [new branch]                gh/davidberard98/392/base -> origin/gh/davidberard98/392/base
2025-12-04T12:26:06.7760667Z  * [new branch]                gh/davidberard98/392/head -> origin/gh/davidberard98/392/head
2025-12-04T12:26:06.7760748Z  * [new branch]                gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig
2025-12-04T12:26:06.7760830Z  * [new branch]                gh/davidberard98/399/base -> origin/gh/davidberard98/399/base
2025-12-04T12:26:06.7760913Z  * [new branch]                gh/davidberard98/399/head -> origin/gh/davidberard98/399/head
2025-12-04T12:26:06.7760993Z  * [new branch]                gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig
2025-12-04T12:26:06.7761069Z  * [new branch]                gh/desertfire/605/base  -> origin/gh/desertfire/605/base
2025-12-04T12:26:06.7761144Z  * [new branch]                gh/desertfire/605/head  -> origin/gh/desertfire/605/head
2025-12-04T12:26:06.7761218Z  * [new branch]                gh/desertfire/605/orig  -> origin/gh/desertfire/605/orig
2025-12-04T12:26:06.7761292Z  * [new branch]                gh/desertfire/606/base  -> origin/gh/desertfire/606/base
2025-12-04T12:26:06.7761367Z  * [new branch]                gh/desertfire/606/head  -> origin/gh/desertfire/606/head
2025-12-04T12:26:06.7761439Z  * [new branch]                gh/desertfire/606/orig  -> origin/gh/desertfire/606/orig
2025-12-04T12:26:06.7761514Z  * [new branch]                gh/desertfire/607/base  -> origin/gh/desertfire/607/base
2025-12-04T12:26:06.7761587Z  * [new branch]                gh/desertfire/607/head  -> origin/gh/desertfire/607/head
2025-12-04T12:26:06.7761660Z  * [new branch]                gh/desertfire/607/orig  -> origin/gh/desertfire/607/orig
2025-12-04T12:26:06.7761734Z  * [new branch]                gh/desertfire/608/base  -> origin/gh/desertfire/608/base
2025-12-04T12:26:06.7761807Z  * [new branch]                gh/desertfire/608/head  -> origin/gh/desertfire/608/head
2025-12-04T12:26:06.7761879Z  * [new branch]                gh/desertfire/608/orig  -> origin/gh/desertfire/608/orig
2025-12-04T12:26:06.7761952Z  * [new branch]                gh/desertfire/609/base  -> origin/gh/desertfire/609/base
2025-12-04T12:26:06.7762025Z  * [new branch]                gh/desertfire/609/head  -> origin/gh/desertfire/609/head
2025-12-04T12:26:06.7762098Z  * [new branch]                gh/desertfire/609/orig  -> origin/gh/desertfire/609/orig
2025-12-04T12:26:06.7762174Z  * [new branch]                gh/desertfire/610/base  -> origin/gh/desertfire/610/base
2025-12-04T12:26:06.7762246Z  * [new branch]                gh/desertfire/610/head  -> origin/gh/desertfire/610/head
2025-12-04T12:26:06.7762318Z  * [new branch]                gh/desertfire/610/orig  -> origin/gh/desertfire/610/orig
2025-12-04T12:26:06.7762392Z  * [new branch]                gh/desertfire/611/base  -> origin/gh/desertfire/611/base
2025-12-04T12:26:06.7762465Z  * [new branch]                gh/desertfire/611/head  -> origin/gh/desertfire/611/head
2025-12-04T12:26:06.7762541Z  * [new branch]                gh/desertfire/611/orig  -> origin/gh/desertfire/611/orig
2025-12-04T12:26:06.7762643Z  * [new branch]                gh/desertfire/612/base  -> origin/gh/desertfire/612/base
2025-12-04T12:26:06.7762718Z  * [new branch]                gh/desertfire/612/head  -> origin/gh/desertfire/612/head
2025-12-04T12:26:06.7762797Z  * [new branch]                gh/desertfire/612/orig  -> origin/gh/desertfire/612/orig
2025-12-04T12:26:06.7762897Z  * [new branch]                gh/desertfire/613/base  -> origin/gh/desertfire/613/base
2025-12-04T12:26:06.7762972Z  * [new branch]                gh/desertfire/613/head  -> origin/gh/desertfire/613/head
2025-12-04T12:26:06.7763051Z  * [new branch]                gh/desertfire/613/orig  -> origin/gh/desertfire/613/orig
2025-12-04T12:26:06.7763124Z  * [new branch]                gh/desertfire/614/base  -> origin/gh/desertfire/614/base
2025-12-04T12:26:06.7763202Z  * [new branch]                gh/desertfire/614/head  -> origin/gh/desertfire/614/head
2025-12-04T12:26:06.7763278Z  * [new branch]                gh/desertfire/614/orig  -> origin/gh/desertfire/614/orig
2025-12-04T12:26:06.7763354Z  * [new branch]                gh/desertfire/615/base  -> origin/gh/desertfire/615/base
2025-12-04T12:26:06.7763431Z  * [new branch]                gh/desertfire/615/head  -> origin/gh/desertfire/615/head
2025-12-04T12:26:06.7763511Z  * [new branch]                gh/desertfire/615/orig  -> origin/gh/desertfire/615/orig
2025-12-04T12:26:06.7763588Z  * [new branch]                gh/desertfire/616/base  -> origin/gh/desertfire/616/base
2025-12-04T12:26:06.7763666Z  * [new branch]                gh/desertfire/616/head  -> origin/gh/desertfire/616/head
2025-12-04T12:26:06.7763743Z  * [new branch]                gh/desertfire/616/orig  -> origin/gh/desertfire/616/orig
2025-12-04T12:26:06.7763817Z  * [new branch]                gh/desertfire/617/base  -> origin/gh/desertfire/617/base
2025-12-04T12:26:06.7763893Z  * [new branch]                gh/desertfire/617/head  -> origin/gh/desertfire/617/head
2025-12-04T12:26:06.7763973Z  * [new branch]                gh/desertfire/617/orig  -> origin/gh/desertfire/617/orig
2025-12-04T12:26:06.7764048Z  * [new branch]                gh/dharakk/1/base       -> origin/gh/dharakk/1/base
2025-12-04T12:26:06.7764125Z  * [new branch]                gh/dharakk/1/head       -> origin/gh/dharakk/1/head
2025-12-04T12:26:06.7764201Z  * [new branch]                gh/drisspg/170/base     -> origin/gh/drisspg/170/base
2025-12-04T12:26:06.7764276Z  * [new branch]                gh/drisspg/170/head     -> origin/gh/drisspg/170/head
2025-12-04T12:26:06.7764354Z  * [new branch]                gh/drisspg/170/orig     -> origin/gh/drisspg/170/orig
2025-12-04T12:26:06.7764427Z  * [new branch]                gh/drisspg/182/base     -> origin/gh/drisspg/182/base
2025-12-04T12:26:06.7764498Z  * [new branch]                gh/drisspg/182/head     -> origin/gh/drisspg/182/head
2025-12-04T12:26:06.7764574Z  * [new branch]                gh/drisspg/183/base     -> origin/gh/drisspg/183/base
2025-12-04T12:26:06.7764646Z  * [new branch]                gh/drisspg/183/head     -> origin/gh/drisspg/183/head
2025-12-04T12:26:06.7764719Z  * [new branch]                gh/drisspg/184/base     -> origin/gh/drisspg/184/base
2025-12-04T12:26:06.7764794Z  * [new branch]                gh/drisspg/184/head     -> origin/gh/drisspg/184/head
2025-12-04T12:26:06.7764867Z  * [new branch]                gh/drisspg/185/base     -> origin/gh/drisspg/185/base
2025-12-04T12:26:06.7764939Z  * [new branch]                gh/drisspg/185/head     -> origin/gh/drisspg/185/head
2025-12-04T12:26:06.7765014Z  * [new branch]                gh/drisspg/194/base     -> origin/gh/drisspg/194/base
2025-12-04T12:26:06.7765085Z  * [new branch]                gh/drisspg/194/head     -> origin/gh/drisspg/194/head
2025-12-04T12:26:06.7765157Z  * [new branch]                gh/drisspg/194/orig     -> origin/gh/drisspg/194/orig
2025-12-04T12:26:06.7765231Z  * [new branch]                gh/drisspg/200/base     -> origin/gh/drisspg/200/base
2025-12-04T12:26:06.7765303Z  * [new branch]                gh/drisspg/200/head     -> origin/gh/drisspg/200/head
2025-12-04T12:26:06.7765400Z  * [new branch]                gh/drisspg/200/orig     -> origin/gh/drisspg/200/orig
2025-12-04T12:26:06.7765477Z  * [new branch]                gh/drisspg/218/base     -> origin/gh/drisspg/218/base
2025-12-04T12:26:06.7765585Z  * [new branch]                gh/drisspg/218/head     -> origin/gh/drisspg/218/head
2025-12-04T12:26:06.7765656Z  * [new branch]                gh/drisspg/218/orig     -> origin/gh/drisspg/218/orig
2025-12-04T12:26:06.7765733Z  * [new branch]                gh/drisspg/219/base     -> origin/gh/drisspg/219/base
2025-12-04T12:26:06.7765805Z  * [new branch]                gh/drisspg/219/head     -> origin/gh/drisspg/219/head
2025-12-04T12:26:06.7765881Z  * [new branch]                gh/drisspg/219/orig     -> origin/gh/drisspg/219/orig
2025-12-04T12:26:06.7765952Z  * [new branch]                gh/drisspg/220/base     -> origin/gh/drisspg/220/base
2025-12-04T12:26:06.7766024Z  * [new branch]                gh/drisspg/220/head     -> origin/gh/drisspg/220/head
2025-12-04T12:26:06.7766102Z  * [new branch]                gh/drisspg/220/orig     -> origin/gh/drisspg/220/orig
2025-12-04T12:26:06.7766174Z  * [new branch]                gh/drisspg/221/base     -> origin/gh/drisspg/221/base
2025-12-04T12:26:06.7766247Z  * [new branch]                gh/drisspg/221/head     -> origin/gh/drisspg/221/head
2025-12-04T12:26:06.7766323Z  * [new branch]                gh/drisspg/221/orig     -> origin/gh/drisspg/221/orig
2025-12-04T12:26:06.7766395Z  * [new branch]                gh/drisspg/222/base     -> origin/gh/drisspg/222/base
2025-12-04T12:26:06.7766467Z  * [new branch]                gh/drisspg/222/head     -> origin/gh/drisspg/222/head
2025-12-04T12:26:06.7766542Z  * [new branch]                gh/drisspg/222/orig     -> origin/gh/drisspg/222/orig
2025-12-04T12:26:06.7766612Z  * [new branch]                gh/drisspg/223/base     -> origin/gh/drisspg/223/base
2025-12-04T12:26:06.7766684Z  * [new branch]                gh/drisspg/223/head     -> origin/gh/drisspg/223/head
2025-12-04T12:26:06.7766808Z  * [new branch]                gh/drisspg/223/orig     -> origin/gh/drisspg/223/orig
2025-12-04T12:26:06.7766882Z  * [new branch]                gh/drisspg/224/base     -> origin/gh/drisspg/224/base
2025-12-04T12:26:06.7766955Z  * [new branch]                gh/drisspg/224/head     -> origin/gh/drisspg/224/head
2025-12-04T12:26:06.7767030Z  * [new branch]                gh/drisspg/224/orig     -> origin/gh/drisspg/224/orig
2025-12-04T12:26:06.7767101Z  * [new branch]                gh/drisspg/225/base     -> origin/gh/drisspg/225/base
2025-12-04T12:26:06.7767173Z  * [new branch]                gh/drisspg/225/head     -> origin/gh/drisspg/225/head
2025-12-04T12:26:06.7767249Z  * [new branch]                gh/drisspg/225/orig     -> origin/gh/drisspg/225/orig
2025-12-04T12:26:06.7767321Z  * [new branch]                gh/drisspg/226/base     -> origin/gh/drisspg/226/base
2025-12-04T12:26:06.7767393Z  * [new branch]                gh/drisspg/226/head     -> origin/gh/drisspg/226/head
2025-12-04T12:26:06.7767469Z  * [new branch]                gh/drisspg/226/orig     -> origin/gh/drisspg/226/orig
2025-12-04T12:26:06.7767540Z  * [new branch]                gh/drisspg/227/base     -> origin/gh/drisspg/227/base
2025-12-04T12:26:06.7767615Z  * [new branch]                gh/drisspg/227/head     -> origin/gh/drisspg/227/head
2025-12-04T12:26:06.7767685Z  * [new branch]                gh/drisspg/227/orig     -> origin/gh/drisspg/227/orig
2025-12-04T12:26:06.7767753Z  * [new branch]                gh/drisspg/228/base     -> origin/gh/drisspg/228/base
2025-12-04T12:26:06.7767824Z  * [new branch]                gh/drisspg/228/head     -> origin/gh/drisspg/228/head
2025-12-04T12:26:06.7767894Z  * [new branch]                gh/drisspg/228/orig     -> origin/gh/drisspg/228/orig
2025-12-04T12:26:06.7767963Z  * [new branch]                gh/drisspg/229/base     -> origin/gh/drisspg/229/base
2025-12-04T12:26:06.7768034Z  * [new branch]                gh/drisspg/229/head     -> origin/gh/drisspg/229/head
2025-12-04T12:26:06.7768152Z  * [new branch]                gh/drisspg/229/orig     -> origin/gh/drisspg/229/orig
2025-12-04T12:26:06.7768223Z  * [new branch]                gh/drisspg/230/base     -> origin/gh/drisspg/230/base
2025-12-04T12:26:06.7768332Z  * [new branch]                gh/drisspg/230/head     -> origin/gh/drisspg/230/head
2025-12-04T12:26:06.7768402Z  * [new branch]                gh/drisspg/230/orig     -> origin/gh/drisspg/230/orig
2025-12-04T12:26:06.7768474Z  * [new branch]                gh/dsjohns2/1/base      -> origin/gh/dsjohns2/1/base
2025-12-04T12:26:06.7768547Z  * [new branch]                gh/dsjohns2/1/head      -> origin/gh/dsjohns2/1/head
2025-12-04T12:26:06.7768628Z  * [new branch]                gh/dzmitry-huba/1/base  -> origin/gh/dzmitry-huba/1/base
2025-12-04T12:26:06.7768705Z  * [new branch]                gh/dzmitry-huba/1/head  -> origin/gh/dzmitry-huba/1/head
2025-12-04T12:26:06.7768787Z  * [new branch]                gh/dzmitry-huba/12/base -> origin/gh/dzmitry-huba/12/base
2025-12-04T12:26:06.7768865Z  * [new branch]                gh/dzmitry-huba/12/head -> origin/gh/dzmitry-huba/12/head
2025-12-04T12:26:06.7768941Z  * [new branch]                gh/dzmitry-huba/12/orig -> origin/gh/dzmitry-huba/12/orig
2025-12-04T12:26:06.7769022Z  * [new branch]                gh/dzmitry-huba/13/base -> origin/gh/dzmitry-huba/13/base
2025-12-04T12:26:06.7769097Z  * [new branch]                gh/dzmitry-huba/13/head -> origin/gh/dzmitry-huba/13/head
2025-12-04T12:26:06.7769172Z  * [new branch]                gh/dzmitry-huba/13/orig -> origin/gh/dzmitry-huba/13/orig
2025-12-04T12:26:06.7769251Z  * [new branch]                gh/dzmitry-huba/14/base -> origin/gh/dzmitry-huba/14/base
2025-12-04T12:26:06.7769326Z  * [new branch]                gh/dzmitry-huba/14/head -> origin/gh/dzmitry-huba/14/head
2025-12-04T12:26:06.7769404Z  * [new branch]                gh/dzmitry-huba/14/orig -> origin/gh/dzmitry-huba/14/orig
2025-12-04T12:26:06.7769480Z  * [new branch]                gh/dzmitry-huba/15/base -> origin/gh/dzmitry-huba/15/base
2025-12-04T12:26:06.7769555Z  * [new branch]                gh/dzmitry-huba/15/head -> origin/gh/dzmitry-huba/15/head
2025-12-04T12:26:06.7769633Z  * [new branch]                gh/dzmitry-huba/15/orig -> origin/gh/dzmitry-huba/15/orig
2025-12-04T12:26:06.7769709Z  * [new branch]                gh/dzmitry-huba/16/base -> origin/gh/dzmitry-huba/16/base
2025-12-04T12:26:06.7769786Z  * [new branch]                gh/dzmitry-huba/16/head -> origin/gh/dzmitry-huba/16/head
2025-12-04T12:26:06.7769868Z  * [new branch]                gh/dzmitry-huba/16/orig -> origin/gh/dzmitry-huba/16/orig
2025-12-04T12:26:06.7769944Z  * [new branch]                gh/dzmitry-huba/17/base -> origin/gh/dzmitry-huba/17/base
2025-12-04T12:26:06.7770020Z  * [new branch]                gh/dzmitry-huba/17/head -> origin/gh/dzmitry-huba/17/head
2025-12-04T12:26:06.7770097Z  * [new branch]                gh/dzmitry-huba/17/orig -> origin/gh/dzmitry-huba/17/orig
2025-12-04T12:26:06.7770174Z  * [new branch]                gh/dzmitry-huba/2/base  -> origin/gh/dzmitry-huba/2/base
2025-12-04T12:26:06.7770250Z  * [new branch]                gh/dzmitry-huba/2/head  -> origin/gh/dzmitry-huba/2/head
2025-12-04T12:26:06.7770328Z  * [new branch]                gh/dzmitry-huba/3/base  -> origin/gh/dzmitry-huba/3/base
2025-12-04T12:26:06.7770405Z  * [new branch]                gh/dzmitry-huba/3/head  -> origin/gh/dzmitry-huba/3/head
2025-12-04T12:26:06.7770482Z  * [new branch]                gh/eellison/808/base    -> origin/gh/eellison/808/base
2025-12-04T12:26:06.7770562Z  * [new branch]                gh/eellison/808/head    -> origin/gh/eellison/808/head
2025-12-04T12:26:06.7770635Z  * [new branch]                gh/eellison/808/orig    -> origin/gh/eellison/808/orig
2025-12-04T12:26:06.7770709Z  * [new branch]                gh/eellison/822/base    -> origin/gh/eellison/822/base
2025-12-04T12:26:06.7770788Z  * [new branch]                gh/eellison/822/head    -> origin/gh/eellison/822/head
2025-12-04T12:26:06.7770891Z  * [new branch]                gh/eellison/822/orig    -> origin/gh/eellison/822/orig
2025-12-04T12:26:06.7770965Z  * [new branch]                gh/eellison/823/base    -> origin/gh/eellison/823/base
2025-12-04T12:26:06.7771067Z  * [new branch]                gh/eellison/823/head    -> origin/gh/eellison/823/head
2025-12-04T12:26:06.7771140Z  * [new branch]                gh/eellison/823/orig    -> origin/gh/eellison/823/orig
2025-12-04T12:26:06.7771215Z  * [new branch]                gh/eellison/862/base    -> origin/gh/eellison/862/base
2025-12-04T12:26:06.7771287Z  * [new branch]                gh/eellison/862/head    -> origin/gh/eellison/862/head
2025-12-04T12:26:06.7771358Z  * [new branch]                gh/eellison/862/orig    -> origin/gh/eellison/862/orig
2025-12-04T12:26:06.7771435Z  * [new branch]                gh/eellison/863/base    -> origin/gh/eellison/863/base
2025-12-04T12:26:06.7771505Z  * [new branch]                gh/eellison/863/head    -> origin/gh/eellison/863/head
2025-12-04T12:26:06.7771578Z  * [new branch]                gh/eellison/863/orig    -> origin/gh/eellison/863/orig
2025-12-04T12:26:06.7771655Z  * [new branch]                gh/eellison/864/base    -> origin/gh/eellison/864/base
2025-12-04T12:26:06.7771728Z  * [new branch]                gh/eellison/864/head    -> origin/gh/eellison/864/head
2025-12-04T12:26:06.7771799Z  * [new branch]                gh/eellison/864/orig    -> origin/gh/eellison/864/orig
2025-12-04T12:26:06.7771874Z  * [new branch]                gh/eellison/865/base    -> origin/gh/eellison/865/base
2025-12-04T12:26:06.7771946Z  * [new branch]                gh/eellison/865/head    -> origin/gh/eellison/865/head
2025-12-04T12:26:06.7772018Z  * [new branch]                gh/eellison/865/orig    -> origin/gh/eellison/865/orig
2025-12-04T12:26:06.7772093Z  * [new branch]                gh/eellison/866/base    -> origin/gh/eellison/866/base
2025-12-04T12:26:06.7772167Z  * [new branch]                gh/eellison/866/head    -> origin/gh/eellison/866/head
2025-12-04T12:26:06.7772243Z  * [new branch]                gh/eellison/866/orig    -> origin/gh/eellison/866/orig
2025-12-04T12:26:06.7772319Z  * [new branch]                gh/eellison/867/base    -> origin/gh/eellison/867/base
2025-12-04T12:26:06.7772394Z  * [new branch]                gh/eellison/867/head    -> origin/gh/eellison/867/head
2025-12-04T12:26:06.7772467Z  * [new branch]                gh/eellison/867/orig    -> origin/gh/eellison/867/orig
2025-12-04T12:26:06.7772545Z  * [new branch]                gh/eellison/868/base    -> origin/gh/eellison/868/base
2025-12-04T12:26:06.7772618Z  * [new branch]                gh/eellison/868/head    -> origin/gh/eellison/868/head
2025-12-04T12:26:06.7772694Z  * [new branch]                gh/eellison/868/orig    -> origin/gh/eellison/868/orig
2025-12-04T12:26:06.7772768Z  * [new branch]                gh/eellison/869/base    -> origin/gh/eellison/869/base
2025-12-04T12:26:06.7772840Z  * [new branch]                gh/eellison/869/head    -> origin/gh/eellison/869/head
2025-12-04T12:26:06.7772919Z  * [new branch]                gh/eellison/869/orig    -> origin/gh/eellison/869/orig
2025-12-04T12:26:06.7772992Z  * [new branch]                gh/eellison/870/base    -> origin/gh/eellison/870/base
2025-12-04T12:26:06.7773067Z  * [new branch]                gh/eellison/870/head    -> origin/gh/eellison/870/head
2025-12-04T12:26:06.7773145Z  * [new branch]                gh/eellison/870/orig    -> origin/gh/eellison/870/orig
2025-12-04T12:26:06.7773218Z  * [new branch]                gh/eellison/871/base    -> origin/gh/eellison/871/base
2025-12-04T12:26:06.7773291Z  * [new branch]                gh/eellison/871/head    -> origin/gh/eellison/871/head
2025-12-04T12:26:06.7773368Z  * [new branch]                gh/eellison/871/orig    -> origin/gh/eellison/871/orig
2025-12-04T12:26:06.7773441Z  * [new branch]                gh/eellison/872/base    -> origin/gh/eellison/872/base
2025-12-04T12:26:06.7773514Z  * [new branch]                gh/eellison/872/head    -> origin/gh/eellison/872/head
2025-12-04T12:26:06.7773616Z  * [new branch]                gh/eellison/872/orig    -> origin/gh/eellison/872/orig
2025-12-04T12:26:06.7773690Z  * [new branch]                gh/eellison/873/base    -> origin/gh/eellison/873/base
2025-12-04T12:26:06.7773797Z  * [new branch]                gh/eellison/873/head    -> origin/gh/eellison/873/head
2025-12-04T12:26:06.7773874Z  * [new branch]                gh/eellison/873/orig    -> origin/gh/eellison/873/orig
2025-12-04T12:26:06.7773948Z  * [new branch]                gh/eellison/874/base    -> origin/gh/eellison/874/base
2025-12-04T12:26:06.7774021Z  * [new branch]                gh/eellison/874/head    -> origin/gh/eellison/874/head
2025-12-04T12:26:06.7774098Z  * [new branch]                gh/eellison/874/orig    -> origin/gh/eellison/874/orig
2025-12-04T12:26:06.7774170Z  * [new branch]                gh/eellison/875/base    -> origin/gh/eellison/875/base
2025-12-04T12:26:06.7774248Z  * [new branch]                gh/eellison/875/head    -> origin/gh/eellison/875/head
2025-12-04T12:26:06.7774322Z  * [new branch]                gh/eellison/875/orig    -> origin/gh/eellison/875/orig
2025-12-04T12:26:06.7774393Z  * [new branch]                gh/eellison/876/base    -> origin/gh/eellison/876/base
2025-12-04T12:26:06.7774471Z  * [new branch]                gh/eellison/876/head    -> origin/gh/eellison/876/head
2025-12-04T12:26:06.7774544Z  * [new branch]                gh/eellison/876/orig    -> origin/gh/eellison/876/orig
2025-12-04T12:26:06.7774614Z  * [new branch]                gh/eellison/877/base    -> origin/gh/eellison/877/base
2025-12-04T12:26:06.7774688Z  * [new branch]                gh/eellison/877/head    -> origin/gh/eellison/877/head
2025-12-04T12:26:06.7774758Z  * [new branch]                gh/eellison/877/orig    -> origin/gh/eellison/877/orig
2025-12-04T12:26:06.7774829Z  * [new branch]                gh/eellison/878/base    -> origin/gh/eellison/878/base
2025-12-04T12:26:06.7774904Z  * [new branch]                gh/eellison/878/head    -> origin/gh/eellison/878/head
2025-12-04T12:26:06.7774975Z  * [new branch]                gh/eellison/878/orig    -> origin/gh/eellison/878/orig
2025-12-04T12:26:06.7775046Z  * [new branch]                gh/eellison/879/base    -> origin/gh/eellison/879/base
2025-12-04T12:26:06.7775121Z  * [new branch]                gh/eellison/879/head    -> origin/gh/eellison/879/head
2025-12-04T12:26:06.7775192Z  * [new branch]                gh/eellison/879/orig    -> origin/gh/eellison/879/orig
2025-12-04T12:26:06.7775263Z  * [new branch]                gh/eellison/880/base    -> origin/gh/eellison/880/base
2025-12-04T12:26:06.7775336Z  * [new branch]                gh/eellison/880/head    -> origin/gh/eellison/880/head
2025-12-04T12:26:06.7775408Z  * [new branch]                gh/eellison/880/orig    -> origin/gh/eellison/880/orig
2025-12-04T12:26:06.7775479Z  * [new branch]                gh/eellison/881/base    -> origin/gh/eellison/881/base
2025-12-04T12:26:06.7775553Z  * [new branch]                gh/eellison/881/head    -> origin/gh/eellison/881/head
2025-12-04T12:26:06.7775625Z  * [new branch]                gh/eellison/881/orig    -> origin/gh/eellison/881/orig
2025-12-04T12:26:06.7775699Z  * [new branch]                gh/eellison/882/base    -> origin/gh/eellison/882/base
2025-12-04T12:26:06.7775772Z  * [new branch]                gh/eellison/882/head    -> origin/gh/eellison/882/head
2025-12-04T12:26:06.7775843Z  * [new branch]                gh/eellison/882/orig    -> origin/gh/eellison/882/orig
2025-12-04T12:26:06.7775920Z  * [new branch]                gh/eellison/883/base    -> origin/gh/eellison/883/base
2025-12-04T12:26:06.7775991Z  * [new branch]                gh/eellison/883/head    -> origin/gh/eellison/883/head
2025-12-04T12:26:06.7776063Z  * [new branch]                gh/eellison/883/orig    -> origin/gh/eellison/883/orig
2025-12-04T12:26:06.7776137Z  * [new branch]                gh/eellison/884/base    -> origin/gh/eellison/884/base
2025-12-04T12:26:06.7776243Z  * [new branch]                gh/eellison/884/head    -> origin/gh/eellison/884/head
2025-12-04T12:26:06.7776315Z  * [new branch]                gh/eellison/884/orig    -> origin/gh/eellison/884/orig
2025-12-04T12:26:06.7776387Z  * [new branch]                gh/etaf/147/base        -> origin/gh/etaf/147/base
2025-12-04T12:26:06.7776479Z  * [new branch]                gh/etaf/147/head        -> origin/gh/etaf/147/head
2025-12-04T12:26:06.7776545Z  * [new branch]                gh/etaf/154/base        -> origin/gh/etaf/154/base
2025-12-04T12:26:06.7776614Z  * [new branch]                gh/etaf/154/head        -> origin/gh/etaf/154/head
2025-12-04T12:26:06.7776679Z  * [new branch]                gh/etaf/154/orig        -> origin/gh/etaf/154/orig
2025-12-04T12:26:06.7776786Z  * [new branch]                gh/etaf/156/base        -> origin/gh/etaf/156/base
2025-12-04T12:26:06.7776856Z  * [new branch]                gh/etaf/156/head        -> origin/gh/etaf/156/head
2025-12-04T12:26:06.7776921Z  * [new branch]                gh/etaf/156/orig        -> origin/gh/etaf/156/orig
2025-12-04T12:26:06.7776987Z  * [new branch]                gh/etaf/157/base        -> origin/gh/etaf/157/base
2025-12-04T12:26:06.7777056Z  * [new branch]                gh/etaf/157/head        -> origin/gh/etaf/157/head
2025-12-04T12:26:06.7777123Z  * [new branch]                gh/etaf/157/orig        -> origin/gh/etaf/157/orig
2025-12-04T12:26:06.7777188Z  * [new branch]                gh/etaf/158/base        -> origin/gh/etaf/158/base
2025-12-04T12:26:06.7777256Z  * [new branch]                gh/etaf/158/head        -> origin/gh/etaf/158/head
2025-12-04T12:26:06.7777321Z  * [new branch]                gh/etaf/158/orig        -> origin/gh/etaf/158/orig
2025-12-04T12:26:06.7777388Z  * [new branch]                gh/etaf/159/base        -> origin/gh/etaf/159/base
2025-12-04T12:26:06.7777453Z  * [new branch]                gh/etaf/159/head        -> origin/gh/etaf/159/head
2025-12-04T12:26:06.7777518Z  * [new branch]                gh/etaf/159/orig        -> origin/gh/etaf/159/orig
2025-12-04T12:26:06.7777588Z  * [new branch]                gh/etaf/160/base        -> origin/gh/etaf/160/base
2025-12-04T12:26:06.7777653Z  * [new branch]                gh/etaf/160/head        -> origin/gh/etaf/160/head
2025-12-04T12:26:06.7777720Z  * [new branch]                gh/etaf/160/orig        -> origin/gh/etaf/160/orig
2025-12-04T12:26:06.7777787Z  * [new branch]                gh/etaf/161/base        -> origin/gh/etaf/161/base
2025-12-04T12:26:06.7777852Z  * [new branch]                gh/etaf/161/head        -> origin/gh/etaf/161/head
2025-12-04T12:26:06.7777918Z  * [new branch]                gh/etaf/161/orig        -> origin/gh/etaf/161/orig
2025-12-04T12:26:06.7777984Z  * [new branch]                gh/etaf/166/base        -> origin/gh/etaf/166/base
2025-12-04T12:26:06.7778049Z  * [new branch]                gh/etaf/166/head        -> origin/gh/etaf/166/head
2025-12-04T12:26:06.7778114Z  * [new branch]                gh/etaf/166/orig        -> origin/gh/etaf/166/orig
2025-12-04T12:26:06.7778182Z  * [new branch]                gh/etaf/167/base        -> origin/gh/etaf/167/base
2025-12-04T12:26:06.7778246Z  * [new branch]                gh/etaf/167/head        -> origin/gh/etaf/167/head
2025-12-04T12:26:06.7778313Z  * [new branch]                gh/etaf/167/orig        -> origin/gh/etaf/167/orig
2025-12-04T12:26:06.7778381Z  * [new branch]                gh/etaf/168/base        -> origin/gh/etaf/168/base
2025-12-04T12:26:06.7778445Z  * [new branch]                gh/etaf/168/head        -> origin/gh/etaf/168/head
2025-12-04T12:26:06.7778509Z  * [new branch]                gh/etaf/168/orig        -> origin/gh/etaf/168/orig
2025-12-04T12:26:06.7778576Z  * [new branch]                gh/etaf/172/base        -> origin/gh/etaf/172/base
2025-12-04T12:26:06.7778641Z  * [new branch]                gh/etaf/172/head        -> origin/gh/etaf/172/head
2025-12-04T12:26:06.7778706Z  * [new branch]                gh/etaf/172/orig        -> origin/gh/etaf/172/orig
2025-12-04T12:26:06.7778808Z  * [new branch]                gh/etaf/173/base        -> origin/gh/etaf/173/base
2025-12-04T12:26:06.7778876Z  * [new branch]                gh/etaf/173/head        -> origin/gh/etaf/173/head
2025-12-04T12:26:06.7778943Z  * [new branch]                gh/etaf/173/orig        -> origin/gh/etaf/173/orig
2025-12-04T12:26:06.7779045Z  * [new branch]                gh/etaf/174/base        -> origin/gh/etaf/174/base
2025-12-04T12:26:06.7779112Z  * [new branch]                gh/etaf/174/head        -> origin/gh/etaf/174/head
2025-12-04T12:26:06.7779179Z  * [new branch]                gh/etaf/175/base        -> origin/gh/etaf/175/base
2025-12-04T12:26:06.7779244Z  * [new branch]                gh/etaf/175/head        -> origin/gh/etaf/175/head
2025-12-04T12:26:06.7779309Z  * [new branch]                gh/etaf/175/orig        -> origin/gh/etaf/175/orig
2025-12-04T12:26:06.7779378Z  * [new branch]                gh/etaf/176/base        -> origin/gh/etaf/176/base
2025-12-04T12:26:06.7779444Z  * [new branch]                gh/etaf/176/head        -> origin/gh/etaf/176/head
2025-12-04T12:26:06.7779510Z  * [new branch]                gh/etaf/176/orig        -> origin/gh/etaf/176/orig
2025-12-04T12:26:06.7779577Z  * [new branch]                gh/etaf/177/base        -> origin/gh/etaf/177/base
2025-12-04T12:26:06.7779643Z  * [new branch]                gh/etaf/177/head        -> origin/gh/etaf/177/head
2025-12-04T12:26:06.7779709Z  * [new branch]                gh/etaf/177/orig        -> origin/gh/etaf/177/orig
2025-12-04T12:26:06.7779777Z  * [new branch]                gh/etaf/178/base        -> origin/gh/etaf/178/base
2025-12-04T12:26:06.7779843Z  * [new branch]                gh/etaf/178/head        -> origin/gh/etaf/178/head
2025-12-04T12:26:06.7779908Z  * [new branch]                gh/etaf/178/orig        -> origin/gh/etaf/178/orig
2025-12-04T12:26:06.7779976Z  * [new branch]                gh/etaf/179/base        -> origin/gh/etaf/179/base
2025-12-04T12:26:06.7780043Z  * [new branch]                gh/etaf/179/head        -> origin/gh/etaf/179/head
2025-12-04T12:26:06.7780109Z  * [new branch]                gh/etaf/179/orig        -> origin/gh/etaf/179/orig
2025-12-04T12:26:06.7780179Z  * [new branch]                gh/etaf/180/base        -> origin/gh/etaf/180/base
2025-12-04T12:26:06.7780247Z  * [new branch]                gh/etaf/180/head        -> origin/gh/etaf/180/head
2025-12-04T12:26:06.7780312Z  * [new branch]                gh/etaf/180/orig        -> origin/gh/etaf/180/orig
2025-12-04T12:26:06.7780396Z  * [new branch]                gh/exclamaforte/1/base  -> origin/gh/exclamaforte/1/base
2025-12-04T12:26:06.7780474Z  * [new branch]                gh/exclamaforte/1/head  -> origin/gh/exclamaforte/1/head
2025-12-04T12:26:06.7780551Z  * [new branch]                gh/exclamaforte/2/base  -> origin/gh/exclamaforte/2/base
2025-12-04T12:26:06.7780629Z  * [new branch]                gh/exclamaforte/2/head  -> origin/gh/exclamaforte/2/head
2025-12-04T12:26:06.7780706Z  * [new branch]                gh/exclamaforte/3/base  -> origin/gh/exclamaforte/3/base
2025-12-04T12:26:06.7780786Z  * [new branch]                gh/exclamaforte/3/head  -> origin/gh/exclamaforte/3/head
2025-12-04T12:26:06.7780862Z  * [new branch]                gh/exclamaforte/4/base  -> origin/gh/exclamaforte/4/base
2025-12-04T12:26:06.7780939Z  * [new branch]                gh/exclamaforte/4/head  -> origin/gh/exclamaforte/4/head
2025-12-04T12:26:06.7781014Z  * [new branch]                gh/ezyang/2374/base     -> origin/gh/ezyang/2374/base
2025-12-04T12:26:06.7781085Z  * [new branch]                gh/ezyang/2374/head     -> origin/gh/ezyang/2374/head
2025-12-04T12:26:06.7781156Z  * [new branch]                gh/ezyang/2374/orig     -> origin/gh/ezyang/2374/orig
2025-12-04T12:26:06.7781229Z  * [new branch]                gh/ezyang/2973/base     -> origin/gh/ezyang/2973/base
2025-12-04T12:26:06.7781299Z  * [new branch]                gh/ezyang/2973/head     -> origin/gh/ezyang/2973/head
2025-12-04T12:26:06.7781393Z  * [new branch]                gh/ezyang/2973/orig     -> origin/gh/ezyang/2973/orig
2025-12-04T12:26:06.7781466Z  * [new branch]                gh/ezyang/2974/base     -> origin/gh/ezyang/2974/base
2025-12-04T12:26:06.7781535Z  * [new branch]                gh/ezyang/2974/head     -> origin/gh/ezyang/2974/head
2025-12-04T12:26:06.7781630Z  * [new branch]                gh/ezyang/2974/orig     -> origin/gh/ezyang/2974/orig
2025-12-04T12:26:06.7781701Z  * [new branch]                gh/ezyang/3131/base     -> origin/gh/ezyang/3131/base
2025-12-04T12:26:06.7781770Z  * [new branch]                gh/ezyang/3131/head     -> origin/gh/ezyang/3131/head
2025-12-04T12:26:06.7781840Z  * [new branch]                gh/ezyang/3131/orig     -> origin/gh/ezyang/3131/orig
2025-12-04T12:26:06.7781912Z  * [new branch]                gh/ezyang/3139/base     -> origin/gh/ezyang/3139/base
2025-12-04T12:26:06.7781980Z  * [new branch]                gh/ezyang/3139/head     -> origin/gh/ezyang/3139/head
2025-12-04T12:26:06.7782050Z  * [new branch]                gh/ezyang/3139/orig     -> origin/gh/ezyang/3139/orig
2025-12-04T12:26:06.7782122Z  * [new branch]                gh/ezyang/3140/base     -> origin/gh/ezyang/3140/base
2025-12-04T12:26:06.7782191Z  * [new branch]                gh/ezyang/3140/head     -> origin/gh/ezyang/3140/head
2025-12-04T12:26:06.7782263Z  * [new branch]                gh/ezyang/3140/orig     -> origin/gh/ezyang/3140/orig
2025-12-04T12:26:06.7782334Z  * [new branch]                gh/ezyang/3143/base     -> origin/gh/ezyang/3143/base
2025-12-04T12:26:06.7782402Z  * [new branch]                gh/ezyang/3143/head     -> origin/gh/ezyang/3143/head
2025-12-04T12:26:06.7782473Z  * [new branch]                gh/ezyang/3143/orig     -> origin/gh/ezyang/3143/orig
2025-12-04T12:26:06.7782541Z  * [new branch]                gh/ezyang/3144/base     -> origin/gh/ezyang/3144/base
2025-12-04T12:26:06.7782609Z  * [new branch]                gh/ezyang/3144/head     -> origin/gh/ezyang/3144/head
2025-12-04T12:26:06.7782682Z  * [new branch]                gh/ezyang/3144/orig     -> origin/gh/ezyang/3144/orig
2025-12-04T12:26:06.7782750Z  * [new branch]                gh/ezyang/3167/base     -> origin/gh/ezyang/3167/base
2025-12-04T12:26:06.7782820Z  * [new branch]                gh/ezyang/3167/head     -> origin/gh/ezyang/3167/head
2025-12-04T12:26:06.7782893Z  * [new branch]                gh/ezyang/3167/orig     -> origin/gh/ezyang/3167/orig
2025-12-04T12:26:06.7782962Z  * [new branch]                gh/ezyang/3173/base     -> origin/gh/ezyang/3173/base
2025-12-04T12:26:06.7783030Z  * [new branch]                gh/ezyang/3173/head     -> origin/gh/ezyang/3173/head
2025-12-04T12:26:06.7783101Z  * [new branch]                gh/ezyang/3173/orig     -> origin/gh/ezyang/3173/orig
2025-12-04T12:26:06.7783169Z  * [new branch]                gh/ezyang/3175/base     -> origin/gh/ezyang/3175/base
2025-12-04T12:26:06.7783237Z  * [new branch]                gh/ezyang/3175/head     -> origin/gh/ezyang/3175/head
2025-12-04T12:26:06.7783309Z  * [new branch]                gh/ezyang/3175/orig     -> origin/gh/ezyang/3175/orig
2025-12-04T12:26:06.7783377Z  * [new branch]                gh/ezyang/3182/base     -> origin/gh/ezyang/3182/base
2025-12-04T12:26:06.7783446Z  * [new branch]                gh/ezyang/3182/head     -> origin/gh/ezyang/3182/head
2025-12-04T12:26:06.7783518Z  * [new branch]                gh/ezyang/3182/orig     -> origin/gh/ezyang/3182/orig
2025-12-04T12:26:06.7783587Z  * [new branch]                gh/ezyang/3185/base     -> origin/gh/ezyang/3185/base
2025-12-04T12:26:06.7783655Z  * [new branch]                gh/ezyang/3185/head     -> origin/gh/ezyang/3185/head
2025-12-04T12:26:06.7783726Z  * [new branch]                gh/ezyang/3185/orig     -> origin/gh/ezyang/3185/orig
2025-12-04T12:26:06.7783794Z  * [new branch]                gh/ezyang/3189/base     -> origin/gh/ezyang/3189/base
2025-12-04T12:26:06.7783862Z  * [new branch]                gh/ezyang/3189/head     -> origin/gh/ezyang/3189/head
2025-12-04T12:26:06.7783958Z  * [new branch]                gh/ezyang/3189/orig     -> origin/gh/ezyang/3189/orig
2025-12-04T12:26:06.7784028Z  * [new branch]                gh/ezyang/3191/base     -> origin/gh/ezyang/3191/base
2025-12-04T12:26:06.7784099Z  * [new branch]                gh/ezyang/3191/head     -> origin/gh/ezyang/3191/head
2025-12-04T12:26:06.7784195Z  * [new branch]                gh/ezyang/3191/orig     -> origin/gh/ezyang/3191/orig
2025-12-04T12:26:06.7784264Z  * [new branch]                gh/ezyang/3192/base     -> origin/gh/ezyang/3192/base
2025-12-04T12:26:06.7784334Z  * [new branch]                gh/ezyang/3192/head     -> origin/gh/ezyang/3192/head
2025-12-04T12:26:06.7784403Z  * [new branch]                gh/ezyang/3192/orig     -> origin/gh/ezyang/3192/orig
2025-12-04T12:26:06.7784471Z  * [new branch]                gh/ezyang/3193/base     -> origin/gh/ezyang/3193/base
2025-12-04T12:26:06.7784542Z  * [new branch]                gh/ezyang/3193/head     -> origin/gh/ezyang/3193/head
2025-12-04T12:26:06.7784611Z  * [new branch]                gh/ezyang/3193/orig     -> origin/gh/ezyang/3193/orig
2025-12-04T12:26:06.7784680Z  * [new branch]                gh/ezyang/3194/base     -> origin/gh/ezyang/3194/base
2025-12-04T12:26:06.7784752Z  * [new branch]                gh/ezyang/3194/head     -> origin/gh/ezyang/3194/head
2025-12-04T12:26:06.7784823Z  * [new branch]                gh/ezyang/3194/orig     -> origin/gh/ezyang/3194/orig
2025-12-04T12:26:06.7784891Z  * [new branch]                gh/ezyang/3195/base     -> origin/gh/ezyang/3195/base
2025-12-04T12:26:06.7784963Z  * [new branch]                gh/ezyang/3195/head     -> origin/gh/ezyang/3195/head
2025-12-04T12:26:06.7796368Z  * [new branch]                gh/ezyang/3195/orig     -> origin/gh/ezyang/3195/orig
2025-12-04T12:26:06.7796460Z  * [new branch]                gh/ezyang/3196/base     -> origin/gh/ezyang/3196/base
2025-12-04T12:26:06.7796540Z  * [new branch]                gh/ezyang/3196/head     -> origin/gh/ezyang/3196/head
2025-12-04T12:26:06.7796621Z  * [new branch]                gh/ezyang/3196/orig     -> origin/gh/ezyang/3196/orig
2025-12-04T12:26:06.7796693Z  * [new branch]                gh/ezyang/3197/base     -> origin/gh/ezyang/3197/base
2025-12-04T12:26:06.7796806Z  * [new branch]                gh/ezyang/3197/head     -> origin/gh/ezyang/3197/head
2025-12-04T12:26:06.7796885Z  * [new branch]                gh/ezyang/3197/orig     -> origin/gh/ezyang/3197/orig
2025-12-04T12:26:06.7796955Z  * [new branch]                gh/ezyang/3198/base     -> origin/gh/ezyang/3198/base
2025-12-04T12:26:06.7797025Z  * [new branch]                gh/ezyang/3198/head     -> origin/gh/ezyang/3198/head
2025-12-04T12:26:06.7797094Z  * [new branch]                gh/ezyang/3198/orig     -> origin/gh/ezyang/3198/orig
2025-12-04T12:26:06.7797163Z  * [new branch]                gh/ezyang/3199/base     -> origin/gh/ezyang/3199/base
2025-12-04T12:26:06.7797234Z  * [new branch]                gh/ezyang/3199/head     -> origin/gh/ezyang/3199/head
2025-12-04T12:26:06.7797306Z  * [new branch]                gh/ezyang/3199/orig     -> origin/gh/ezyang/3199/orig
2025-12-04T12:26:06.7797377Z  * [new branch]                gh/ezyang/3200/base     -> origin/gh/ezyang/3200/base
2025-12-04T12:26:06.7797446Z  * [new branch]                gh/ezyang/3200/head     -> origin/gh/ezyang/3200/head
2025-12-04T12:26:06.7797516Z  * [new branch]                gh/ezyang/3200/orig     -> origin/gh/ezyang/3200/orig
2025-12-04T12:26:06.7797585Z  * [new branch]                gh/ezyang/3201/base     -> origin/gh/ezyang/3201/base
2025-12-04T12:26:06.7797655Z  * [new branch]                gh/ezyang/3201/head     -> origin/gh/ezyang/3201/head
2025-12-04T12:26:06.7797724Z  * [new branch]                gh/ezyang/3201/orig     -> origin/gh/ezyang/3201/orig
2025-12-04T12:26:06.7797797Z  * [new branch]                gh/ezyang/3202/base     -> origin/gh/ezyang/3202/base
2025-12-04T12:26:06.7797866Z  * [new branch]                gh/ezyang/3202/head     -> origin/gh/ezyang/3202/head
2025-12-04T12:26:06.7798006Z  * [new branch]                gh/ezyang/3202/orig     -> origin/gh/ezyang/3202/orig
2025-12-04T12:26:06.7798080Z  * [new branch]                gh/ezyang/3203/base     -> origin/gh/ezyang/3203/base
2025-12-04T12:26:06.7798150Z  * [new branch]                gh/ezyang/3203/head     -> origin/gh/ezyang/3203/head
2025-12-04T12:26:06.7798264Z  * [new branch]                gh/ezyang/3203/orig     -> origin/gh/ezyang/3203/orig
2025-12-04T12:26:06.7798340Z  * [new branch]                gh/ezyang/3204/base     -> origin/gh/ezyang/3204/base
2025-12-04T12:26:06.7798409Z  * [new branch]                gh/ezyang/3204/head     -> origin/gh/ezyang/3204/head
2025-12-04T12:26:06.7798478Z  * [new branch]                gh/ezyang/3204/orig     -> origin/gh/ezyang/3204/orig
2025-12-04T12:26:06.7798553Z  * [new branch]                gh/ezyang/3205/base     -> origin/gh/ezyang/3205/base
2025-12-04T12:26:06.7798625Z  * [new branch]                gh/ezyang/3205/head     -> origin/gh/ezyang/3205/head
2025-12-04T12:26:06.7798702Z  * [new branch]                gh/ezyang/3205/orig     -> origin/gh/ezyang/3205/orig
2025-12-04T12:26:06.7798779Z  * [new branch]                gh/ezyang/3206/base     -> origin/gh/ezyang/3206/base
2025-12-04T12:26:06.7798849Z  * [new branch]                gh/ezyang/3206/head     -> origin/gh/ezyang/3206/head
2025-12-04T12:26:06.7798921Z  * [new branch]                gh/ezyang/3206/orig     -> origin/gh/ezyang/3206/orig
2025-12-04T12:26:06.7798995Z  * [new branch]                gh/ezyang/3207/base     -> origin/gh/ezyang/3207/base
2025-12-04T12:26:06.7799064Z  * [new branch]                gh/ezyang/3207/head     -> origin/gh/ezyang/3207/head
2025-12-04T12:26:06.7799133Z  * [new branch]                gh/ezyang/3207/orig     -> origin/gh/ezyang/3207/orig
2025-12-04T12:26:06.7799201Z  * [new branch]                gh/ezyang/3208/base     -> origin/gh/ezyang/3208/base
2025-12-04T12:26:06.7799269Z  * [new branch]                gh/ezyang/3208/head     -> origin/gh/ezyang/3208/head
2025-12-04T12:26:06.7799342Z  * [new branch]                gh/ezyang/3208/orig     -> origin/gh/ezyang/3208/orig
2025-12-04T12:26:06.7799416Z  * [new branch]                gh/ezyang/3209/base     -> origin/gh/ezyang/3209/base
2025-12-04T12:26:06.7799489Z  * [new branch]                gh/ezyang/3209/head     -> origin/gh/ezyang/3209/head
2025-12-04T12:26:06.7799567Z  * [new branch]                gh/ezyang/3209/orig     -> origin/gh/ezyang/3209/orig
2025-12-04T12:26:06.7799645Z  * [new branch]                gh/fadara01/3/base      -> origin/gh/fadara01/3/base
2025-12-04T12:26:06.7799720Z  * [new branch]                gh/fadara01/3/head      -> origin/gh/fadara01/3/head
2025-12-04T12:26:06.7799795Z  * [new branch]                gh/fadara01/3/orig      -> origin/gh/fadara01/3/orig
2025-12-04T12:26:06.7799865Z  * [new branch]                gh/fadara01/5/base      -> origin/gh/fadara01/5/base
2025-12-04T12:26:06.7799935Z  * [new branch]                gh/fadara01/5/head      -> origin/gh/fadara01/5/head
2025-12-04T12:26:06.7800008Z  * [new branch]                gh/fadara01/5/orig      -> origin/gh/fadara01/5/orig
2025-12-04T12:26:06.7800077Z  * [new branch]                gh/fadara01/6/base      -> origin/gh/fadara01/6/base
2025-12-04T12:26:06.7800146Z  * [new branch]                gh/fadara01/6/head      -> origin/gh/fadara01/6/head
2025-12-04T12:26:06.7800222Z  * [new branch]                gh/fadara01/6/orig      -> origin/gh/fadara01/6/orig
2025-12-04T12:26:06.7800292Z  * [new branch]                gh/fadara01/7/base      -> origin/gh/fadara01/7/base
2025-12-04T12:26:06.7800360Z  * [new branch]                gh/fadara01/7/head      -> origin/gh/fadara01/7/head
2025-12-04T12:26:06.7800429Z  * [new branch]                gh/fadara01/7/orig      -> origin/gh/fadara01/7/orig
2025-12-04T12:26:06.7800499Z  * [new branch]                gh/fadara01/8/base      -> origin/gh/fadara01/8/base
2025-12-04T12:26:06.7800567Z  * [new branch]                gh/fadara01/8/head      -> origin/gh/fadara01/8/head
2025-12-04T12:26:06.7800665Z  * [new branch]                gh/fadara01/8/orig      -> origin/gh/fadara01/8/orig
2025-12-04T12:26:06.7800735Z  * [new branch]                gh/fadara01/9/base      -> origin/gh/fadara01/9/base
2025-12-04T12:26:06.7800806Z  * [new branch]                gh/fadara01/9/head      -> origin/gh/fadara01/9/head
2025-12-04T12:26:06.7800905Z  * [new branch]                gh/fadara01/9/orig      -> origin/gh/fadara01/9/orig
2025-12-04T12:26:06.7800976Z  * [new branch]                gh/fduwjj/182/base      -> origin/gh/fduwjj/182/base
2025-12-04T12:26:06.7801053Z  * [new branch]                gh/fduwjj/182/head      -> origin/gh/fduwjj/182/head
2025-12-04T12:26:06.7801123Z  * [new branch]                gh/fduwjj/182/orig      -> origin/gh/fduwjj/182/orig
2025-12-04T12:26:06.7801193Z  * [new branch]                gh/fduwjj/211/base      -> origin/gh/fduwjj/211/base
2025-12-04T12:26:06.7801271Z  * [new branch]                gh/fduwjj/211/head      -> origin/gh/fduwjj/211/head
2025-12-04T12:26:06.7801344Z  * [new branch]                gh/fduwjj/211/orig      -> origin/gh/fduwjj/211/orig
2025-12-04T12:26:06.7801414Z  * [new branch]                gh/fduwjj/212/base      -> origin/gh/fduwjj/212/base
2025-12-04T12:26:06.7801485Z  * [new branch]                gh/fduwjj/212/head      -> origin/gh/fduwjj/212/head
2025-12-04T12:26:06.7801556Z  * [new branch]                gh/fduwjj/212/orig      -> origin/gh/fduwjj/212/orig
2025-12-04T12:26:06.7801630Z  * [new branch]                gh/fduwjj/213/base      -> origin/gh/fduwjj/213/base
2025-12-04T12:26:06.7801699Z  * [new branch]                gh/fduwjj/213/head      -> origin/gh/fduwjj/213/head
2025-12-04T12:26:06.7801772Z  * [new branch]                gh/fduwjj/213/orig      -> origin/gh/fduwjj/213/orig
2025-12-04T12:26:06.7801841Z  * [new branch]                gh/fduwjj/226/base      -> origin/gh/fduwjj/226/base
2025-12-04T12:26:06.7801909Z  * [new branch]                gh/fduwjj/226/head      -> origin/gh/fduwjj/226/head
2025-12-04T12:26:06.7801982Z  * [new branch]                gh/fduwjj/226/orig      -> origin/gh/fduwjj/226/orig
2025-12-04T12:26:06.7802051Z  * [new branch]                gh/fduwjj/229/base      -> origin/gh/fduwjj/229/base
2025-12-04T12:26:06.7802119Z  * [new branch]                gh/fduwjj/229/head      -> origin/gh/fduwjj/229/head
2025-12-04T12:26:06.7802193Z  * [new branch]                gh/fduwjj/229/orig      -> origin/gh/fduwjj/229/orig
2025-12-04T12:26:06.7802262Z  * [new branch]                gh/fduwjj/233/base      -> origin/gh/fduwjj/233/base
2025-12-04T12:26:06.7802331Z  * [new branch]                gh/fduwjj/233/head      -> origin/gh/fduwjj/233/head
2025-12-04T12:26:06.7802402Z  * [new branch]                gh/fduwjj/233/orig      -> origin/gh/fduwjj/233/orig
2025-12-04T12:26:06.7802471Z  * [new branch]                gh/fduwjj/234/base      -> origin/gh/fduwjj/234/base
2025-12-04T12:26:06.7802540Z  * [new branch]                gh/fduwjj/234/head      -> origin/gh/fduwjj/234/head
2025-12-04T12:26:06.7802612Z  * [new branch]                gh/fduwjj/234/orig      -> origin/gh/fduwjj/234/orig
2025-12-04T12:26:06.7802681Z  * [new branch]                gh/fduwjj/235/base      -> origin/gh/fduwjj/235/base
2025-12-04T12:26:06.7802751Z  * [new branch]                gh/fduwjj/235/head      -> origin/gh/fduwjj/235/head
2025-12-04T12:26:06.7802822Z  * [new branch]                gh/fduwjj/235/orig      -> origin/gh/fduwjj/235/orig
2025-12-04T12:26:06.7802890Z  * [new branch]                gh/fduwjj/236/base      -> origin/gh/fduwjj/236/base
2025-12-04T12:26:06.7802962Z  * [new branch]                gh/fduwjj/236/head      -> origin/gh/fduwjj/236/head
2025-12-04T12:26:06.7803031Z  * [new branch]                gh/fduwjj/236/orig      -> origin/gh/fduwjj/236/orig
2025-12-04T12:26:06.7803100Z  * [new branch]                gh/fduwjj/237/base      -> origin/gh/fduwjj/237/base
2025-12-04T12:26:06.7803170Z  * [new branch]                gh/fduwjj/237/head      -> origin/gh/fduwjj/237/head
2025-12-04T12:26:06.7803263Z  * [new branch]                gh/fduwjj/237/orig      -> origin/gh/fduwjj/237/orig
2025-12-04T12:26:06.7803333Z  * [new branch]                gh/fduwjj/238/base      -> origin/gh/fduwjj/238/base
2025-12-04T12:26:06.7803402Z  * [new branch]                gh/fduwjj/238/head      -> origin/gh/fduwjj/238/head
2025-12-04T12:26:06.7803497Z  * [new branch]                gh/fduwjj/238/orig      -> origin/gh/fduwjj/238/orig
2025-12-04T12:26:06.7803565Z  * [new branch]                gh/fduwjj/239/base      -> origin/gh/fduwjj/239/base
2025-12-04T12:26:06.7803636Z  * [new branch]                gh/fduwjj/239/head      -> origin/gh/fduwjj/239/head
2025-12-04T12:26:06.7803706Z  * [new branch]                gh/fduwjj/239/orig      -> origin/gh/fduwjj/239/orig
2025-12-04T12:26:06.7803775Z  * [new branch]                gh/fegin/332/base       -> origin/gh/fegin/332/base
2025-12-04T12:26:06.7803848Z  * [new branch]                gh/fegin/332/head       -> origin/gh/fegin/332/head
2025-12-04T12:26:06.7803915Z  * [new branch]                gh/fegin/332/orig       -> origin/gh/fegin/332/orig
2025-12-04T12:26:06.7803983Z  * [new branch]                gh/fegin/333/base       -> origin/gh/fegin/333/base
2025-12-04T12:26:06.7804052Z  * [new branch]                gh/fegin/333/head       -> origin/gh/fegin/333/head
2025-12-04T12:26:06.7804119Z  * [new branch]                gh/fegin/333/orig       -> origin/gh/fegin/333/orig
2025-12-04T12:26:06.7804185Z  * [new branch]                gh/fegin/334/base       -> origin/gh/fegin/334/base
2025-12-04T12:26:06.7804252Z  * [new branch]                gh/fegin/334/head       -> origin/gh/fegin/334/head
2025-12-04T12:26:06.7804318Z  * [new branch]                gh/fegin/334/orig       -> origin/gh/fegin/334/orig
2025-12-04T12:26:06.7804389Z  * [new branch]                gh/fegin/335/base       -> origin/gh/fegin/335/base
2025-12-04T12:26:06.7804457Z  * [new branch]                gh/fegin/335/head       -> origin/gh/fegin/335/head
2025-12-04T12:26:06.7804525Z  * [new branch]                gh/fegin/335/orig       -> origin/gh/fegin/335/orig
2025-12-04T12:26:06.7804597Z  * [new branch]                gh/fffrog/160/base      -> origin/gh/fffrog/160/base
2025-12-04T12:26:06.7804666Z  * [new branch]                gh/fffrog/160/head      -> origin/gh/fffrog/160/head
2025-12-04T12:26:06.7804736Z  * [new branch]                gh/fffrog/177/base      -> origin/gh/fffrog/177/base
2025-12-04T12:26:06.7804808Z  * [new branch]                gh/fffrog/177/head      -> origin/gh/fffrog/177/head
2025-12-04T12:26:06.7804876Z  * [new branch]                gh/fffrog/177/orig      -> origin/gh/fffrog/177/orig
2025-12-04T12:26:06.7804944Z  * [new branch]                gh/fffrog/178/base      -> origin/gh/fffrog/178/base
2025-12-04T12:26:06.7805016Z  * [new branch]                gh/fffrog/178/head      -> origin/gh/fffrog/178/head
2025-12-04T12:26:06.7805084Z  * [new branch]                gh/fffrog/178/orig      -> origin/gh/fffrog/178/orig
2025-12-04T12:26:06.7805152Z  * [new branch]                gh/fffrog/181/base      -> origin/gh/fffrog/181/base
2025-12-04T12:26:06.7805225Z  * [new branch]                gh/fffrog/181/head      -> origin/gh/fffrog/181/head
2025-12-04T12:26:06.7805292Z  * [new branch]                gh/fffrog/181/orig      -> origin/gh/fffrog/181/orig
2025-12-04T12:26:06.7805362Z  * [new branch]                gh/fffrog/183/base      -> origin/gh/fffrog/183/base
2025-12-04T12:26:06.7805431Z  * [new branch]                gh/fffrog/183/head      -> origin/gh/fffrog/183/head
2025-12-04T12:26:06.7805498Z  * [new branch]                gh/fffrog/183/orig      -> origin/gh/fffrog/183/orig
2025-12-04T12:26:06.7805567Z  * [new branch]                gh/fxdawnn/10/base      -> origin/gh/fxdawnn/10/base
2025-12-04T12:26:06.7805639Z  * [new branch]                gh/fxdawnn/10/head      -> origin/gh/fxdawnn/10/head
2025-12-04T12:26:06.7805708Z  * [new branch]                gh/fxdawnn/10/orig      -> origin/gh/fxdawnn/10/orig
2025-12-04T12:26:06.7805778Z  * [new branch]                gh/fxdawnn/11/base      -> origin/gh/fxdawnn/11/base
2025-12-04T12:26:06.7805873Z  * [new branch]                gh/fxdawnn/11/head      -> origin/gh/fxdawnn/11/head
2025-12-04T12:26:06.7805941Z  * [new branch]                gh/fxdawnn/11/orig      -> origin/gh/fxdawnn/11/orig
2025-12-04T12:26:06.7806041Z  * [new branch]                gh/fxdawnn/12/base      -> origin/gh/fxdawnn/12/base
2025-12-04T12:26:06.7806113Z  * [new branch]                gh/fxdawnn/12/head      -> origin/gh/fxdawnn/12/head
2025-12-04T12:26:06.7806181Z  * [new branch]                gh/fxdawnn/12/orig      -> origin/gh/fxdawnn/12/orig
2025-12-04T12:26:06.7806252Z  * [new branch]                gh/fxdawnn/13/base      -> origin/gh/fxdawnn/13/base
2025-12-04T12:26:06.7806320Z  * [new branch]                gh/fxdawnn/13/head      -> origin/gh/fxdawnn/13/head
2025-12-04T12:26:06.7806388Z  * [new branch]                gh/fxdawnn/13/orig      -> origin/gh/fxdawnn/13/orig
2025-12-04T12:26:06.7806460Z  * [new branch]                gh/fxdawnn/14/base      -> origin/gh/fxdawnn/14/base
2025-12-04T12:26:06.7806530Z  * [new branch]                gh/fxdawnn/14/head      -> origin/gh/fxdawnn/14/head
2025-12-04T12:26:06.7806600Z  * [new branch]                gh/fxdawnn/14/orig      -> origin/gh/fxdawnn/14/orig
2025-12-04T12:26:06.7806673Z  * [new branch]                gh/fxdawnn/15/base      -> origin/gh/fxdawnn/15/base
2025-12-04T12:26:06.7806846Z  * [new branch]                gh/fxdawnn/15/head      -> origin/gh/fxdawnn/15/head
2025-12-04T12:26:06.7806918Z  * [new branch]                gh/fxdawnn/15/orig      -> origin/gh/fxdawnn/15/orig
2025-12-04T12:26:06.7806991Z  * [new branch]                gh/fxdawnn/6/base       -> origin/gh/fxdawnn/6/base
2025-12-04T12:26:06.7807061Z  * [new branch]                gh/fxdawnn/6/head       -> origin/gh/fxdawnn/6/head
2025-12-04T12:26:06.7807129Z  * [new branch]                gh/fxdawnn/6/orig       -> origin/gh/fxdawnn/6/orig
2025-12-04T12:26:06.7807200Z  * [new branch]                gh/fxdawnn/7/base       -> origin/gh/fxdawnn/7/base
2025-12-04T12:26:06.7807268Z  * [new branch]                gh/fxdawnn/7/head       -> origin/gh/fxdawnn/7/head
2025-12-04T12:26:06.7807335Z  * [new branch]                gh/fxdawnn/7/orig       -> origin/gh/fxdawnn/7/orig
2025-12-04T12:26:06.7807403Z  * [new branch]                gh/fxdawnn/9/base       -> origin/gh/fxdawnn/9/base
2025-12-04T12:26:06.7807472Z  * [new branch]                gh/fxdawnn/9/head       -> origin/gh/fxdawnn/9/head
2025-12-04T12:26:06.7807540Z  * [new branch]                gh/fxdawnn/9/orig       -> origin/gh/fxdawnn/9/orig
2025-12-04T12:26:06.7807608Z  * [new branch]                gh/galv/1/base          -> origin/gh/galv/1/base
2025-12-04T12:26:06.7807673Z  * [new branch]                gh/galv/1/head          -> origin/gh/galv/1/head
2025-12-04T12:26:06.7807737Z  * [new branch]                gh/galv/1/orig          -> origin/gh/galv/1/orig
2025-12-04T12:26:06.7807803Z  * [new branch]                gh/galv/2/base          -> origin/gh/galv/2/base
2025-12-04T12:26:06.7807869Z  * [new branch]                gh/galv/2/head          -> origin/gh/galv/2/head
2025-12-04T12:26:06.7807934Z  * [new branch]                gh/galv/2/orig          -> origin/gh/galv/2/orig
2025-12-04T12:26:06.7807997Z  * [new branch]                gh/galv/3/base          -> origin/gh/galv/3/base
2025-12-04T12:26:06.7808061Z  * [new branch]                gh/galv/3/head          -> origin/gh/galv/3/head
2025-12-04T12:26:06.7808125Z  * [new branch]                gh/galv/3/orig          -> origin/gh/galv/3/orig
2025-12-04T12:26:06.7808202Z  * [new branch]                gh/guangyey/134/base    -> origin/gh/guangyey/134/base
2025-12-04T12:26:06.7808277Z  * [new branch]                gh/guangyey/134/head    -> origin/gh/guangyey/134/head
2025-12-04T12:26:06.7808350Z  * [new branch]                gh/guangyey/134/orig    -> origin/gh/guangyey/134/orig
2025-12-04T12:26:06.7808421Z  * [new branch]                gh/guangyey/163/base    -> origin/gh/guangyey/163/base
2025-12-04T12:26:06.7808530Z  * [new branch]                gh/guangyey/163/head    -> origin/gh/guangyey/163/head
2025-12-04T12:26:06.7808603Z  * [new branch]                gh/guangyey/163/orig    -> origin/gh/guangyey/163/orig
2025-12-04T12:26:06.7808673Z  * [new branch]                gh/guangyey/168/base    -> origin/gh/guangyey/168/base
2025-12-04T12:26:06.7808777Z  * [new branch]                gh/guangyey/168/head    -> origin/gh/guangyey/168/head
2025-12-04T12:26:06.7808850Z  * [new branch]                gh/guangyey/168/orig    -> origin/gh/guangyey/168/orig
2025-12-04T12:26:06.7808920Z  * [new branch]                gh/guangyey/169/base    -> origin/gh/guangyey/169/base
2025-12-04T12:26:06.7808991Z  * [new branch]                gh/guangyey/169/head    -> origin/gh/guangyey/169/head
2025-12-04T12:26:06.7809063Z  * [new branch]                gh/guangyey/169/orig    -> origin/gh/guangyey/169/orig
2025-12-04T12:26:06.7809134Z  * [new branch]                gh/guangyey/170/base    -> origin/gh/guangyey/170/base
2025-12-04T12:26:06.7809206Z  * [new branch]                gh/guangyey/170/head    -> origin/gh/guangyey/170/head
2025-12-04T12:26:06.7809278Z  * [new branch]                gh/guangyey/170/orig    -> origin/gh/guangyey/170/orig
2025-12-04T12:26:06.7809348Z  * [new branch]                gh/guangyey/171/base    -> origin/gh/guangyey/171/base
2025-12-04T12:26:06.7809420Z  * [new branch]                gh/guangyey/171/head    -> origin/gh/guangyey/171/head
2025-12-04T12:26:06.7809493Z  * [new branch]                gh/guangyey/171/orig    -> origin/gh/guangyey/171/orig
2025-12-04T12:26:06.7809564Z  * [new branch]                gh/guangyey/178/base    -> origin/gh/guangyey/178/base
2025-12-04T12:26:06.7809636Z  * [new branch]                gh/guangyey/178/head    -> origin/gh/guangyey/178/head
2025-12-04T12:26:06.7809706Z  * [new branch]                gh/guangyey/178/orig    -> origin/gh/guangyey/178/orig
2025-12-04T12:26:06.7809776Z  * [new branch]                gh/guangyey/182/base    -> origin/gh/guangyey/182/base
2025-12-04T12:26:06.7809850Z  * [new branch]                gh/guangyey/182/head    -> origin/gh/guangyey/182/head
2025-12-04T12:26:06.7809920Z  * [new branch]                gh/guangyey/182/orig    -> origin/gh/guangyey/182/orig
2025-12-04T12:26:06.7809990Z  * [new branch]                gh/guangyey/183/base    -> origin/gh/guangyey/183/base
2025-12-04T12:26:06.7810064Z  * [new branch]                gh/guangyey/183/head    -> origin/gh/guangyey/183/head
2025-12-04T12:26:06.7810136Z  * [new branch]                gh/guangyey/183/orig    -> origin/gh/guangyey/183/orig
2025-12-04T12:26:06.7810207Z  * [new branch]                gh/guangyey/185/base    -> origin/gh/guangyey/185/base
2025-12-04T12:26:06.7810281Z  * [new branch]                gh/guangyey/185/head    -> origin/gh/guangyey/185/head
2025-12-04T12:26:06.7810352Z  * [new branch]                gh/guangyey/185/orig    -> origin/gh/guangyey/185/orig
2025-12-04T12:26:06.7810421Z  * [new branch]                gh/guangyey/186/base    -> origin/gh/guangyey/186/base
2025-12-04T12:26:06.7810497Z  * [new branch]                gh/guangyey/186/head    -> origin/gh/guangyey/186/head
2025-12-04T12:26:06.7810568Z  * [new branch]                gh/guangyey/186/orig    -> origin/gh/guangyey/186/orig
2025-12-04T12:26:06.7810639Z  * [new branch]                gh/guangyey/187/base    -> origin/gh/guangyey/187/base
2025-12-04T12:26:06.7810714Z  * [new branch]                gh/guangyey/187/head    -> origin/gh/guangyey/187/head
2025-12-04T12:26:06.7810785Z  * [new branch]                gh/guangyey/187/orig    -> origin/gh/guangyey/187/orig
2025-12-04T12:26:06.7810856Z  * [new branch]                gh/guangyey/188/base    -> origin/gh/guangyey/188/base
2025-12-04T12:26:06.7810928Z  * [new branch]                gh/guangyey/188/head    -> origin/gh/guangyey/188/head
2025-12-04T12:26:06.7810999Z  * [new branch]                gh/guangyey/188/orig    -> origin/gh/guangyey/188/orig
2025-12-04T12:26:06.7811069Z  * [new branch]                gh/guangyey/190/base    -> origin/gh/guangyey/190/base
2025-12-04T12:26:06.7811162Z  * [new branch]                gh/guangyey/190/head    -> origin/gh/guangyey/190/head
2025-12-04T12:26:06.7811234Z  * [new branch]                gh/guangyey/190/orig    -> origin/gh/guangyey/190/orig
2025-12-04T12:26:06.7811307Z  * [new branch]                gh/guangyey/208/base    -> origin/gh/guangyey/208/base
2025-12-04T12:26:06.7811454Z  * [new branch]                gh/guangyey/208/head    -> origin/gh/guangyey/208/head
2025-12-04T12:26:06.7811525Z  * [new branch]                gh/guangyey/208/orig    -> origin/gh/guangyey/208/orig
2025-12-04T12:26:06.7811597Z  * [new branch]                gh/guangyey/228/base    -> origin/gh/guangyey/228/base
2025-12-04T12:26:06.7811668Z  * [new branch]                gh/guangyey/228/head    -> origin/gh/guangyey/228/head
2025-12-04T12:26:06.7811738Z  * [new branch]                gh/guangyey/228/orig    -> origin/gh/guangyey/228/orig
2025-12-04T12:26:06.7811810Z  * [new branch]                gh/guangyey/230/base    -> origin/gh/guangyey/230/base
2025-12-04T12:26:06.7811884Z  * [new branch]                gh/guangyey/230/head    -> origin/gh/guangyey/230/head
2025-12-04T12:26:06.7811954Z  * [new branch]                gh/guangyey/230/orig    -> origin/gh/guangyey/230/orig
2025-12-04T12:26:06.7812028Z  * [new branch]                gh/guangyey/231/base    -> origin/gh/guangyey/231/base
2025-12-04T12:26:06.7812098Z  * [new branch]                gh/guangyey/231/head    -> origin/gh/guangyey/231/head
2025-12-04T12:26:06.7812168Z  * [new branch]                gh/guangyey/231/orig    -> origin/gh/guangyey/231/orig
2025-12-04T12:26:06.7812242Z  * [new branch]                gh/guangyey/232/base    -> origin/gh/guangyey/232/base
2025-12-04T12:26:06.7812312Z  * [new branch]                gh/guangyey/232/head    -> origin/gh/guangyey/232/head
2025-12-04T12:26:06.7812382Z  * [new branch]                gh/guangyey/232/orig    -> origin/gh/guangyey/232/orig
2025-12-04T12:26:06.7812453Z  * [new branch]                gh/guangyey/233/base    -> origin/gh/guangyey/233/base
2025-12-04T12:26:06.7812525Z  * [new branch]                gh/guangyey/233/head    -> origin/gh/guangyey/233/head
2025-12-04T12:26:06.7812596Z  * [new branch]                gh/guangyey/233/orig    -> origin/gh/guangyey/233/orig
2025-12-04T12:26:06.7812668Z  * [new branch]                gh/guangyey/234/base    -> origin/gh/guangyey/234/base
2025-12-04T12:26:06.7812739Z  * [new branch]                gh/guangyey/234/head    -> origin/gh/guangyey/234/head
2025-12-04T12:26:06.7812811Z  * [new branch]                gh/guangyey/234/orig    -> origin/gh/guangyey/234/orig
2025-12-04T12:26:06.7812881Z  * [new branch]                gh/guangyey/235/base    -> origin/gh/guangyey/235/base
2025-12-04T12:26:06.7812952Z  * [new branch]                gh/guangyey/235/head    -> origin/gh/guangyey/235/head
2025-12-04T12:26:06.7813024Z  * [new branch]                gh/guangyey/235/orig    -> origin/gh/guangyey/235/orig
2025-12-04T12:26:06.7813095Z  * [new branch]                gh/guangyey/236/base    -> origin/gh/guangyey/236/base
2025-12-04T12:26:06.7813167Z  * [new branch]                gh/guangyey/236/head    -> origin/gh/guangyey/236/head
2025-12-04T12:26:06.7813239Z  * [new branch]                gh/guangyey/236/orig    -> origin/gh/guangyey/236/orig
2025-12-04T12:26:06.7813310Z  * [new branch]                gh/guangyey/237/base    -> origin/gh/guangyey/237/base
2025-12-04T12:26:06.7813380Z  * [new branch]                gh/guangyey/237/head    -> origin/gh/guangyey/237/head
2025-12-04T12:26:06.7813453Z  * [new branch]                gh/guangyey/237/orig    -> origin/gh/guangyey/237/orig
2025-12-04T12:26:06.7813526Z  * [new branch]                gh/guangyey/238/base    -> origin/gh/guangyey/238/base
2025-12-04T12:26:06.7813596Z  * [new branch]                gh/guangyey/238/head    -> origin/gh/guangyey/238/head
2025-12-04T12:26:06.7813671Z  * [new branch]                gh/guangyey/239/base    -> origin/gh/guangyey/239/base
2025-12-04T12:26:06.7813740Z  * [new branch]                gh/guangyey/239/head    -> origin/gh/guangyey/239/head
2025-12-04T12:26:06.7813897Z  * [new branch]                gh/guangyey/239/orig    -> origin/gh/guangyey/239/orig
2025-12-04T12:26:06.7813971Z  * [new branch]                gh/guangyey/240/base    -> origin/gh/guangyey/240/base
2025-12-04T12:26:06.7814070Z  * [new branch]                gh/guangyey/240/head    -> origin/gh/guangyey/240/head
2025-12-04T12:26:06.7814142Z  * [new branch]                gh/guangyey/240/orig    -> origin/gh/guangyey/240/orig
2025-12-04T12:26:06.7814214Z  * [new branch]                gh/guangyey/241/base    -> origin/gh/guangyey/241/base
2025-12-04T12:26:06.7814284Z  * [new branch]                gh/guangyey/241/head    -> origin/gh/guangyey/241/head
2025-12-04T12:26:06.7814355Z  * [new branch]                gh/guangyey/241/orig    -> origin/gh/guangyey/241/orig
2025-12-04T12:26:06.7814426Z  * [new branch]                gh/guangyey/242/base    -> origin/gh/guangyey/242/base
2025-12-04T12:26:06.7814497Z  * [new branch]                gh/guangyey/242/head    -> origin/gh/guangyey/242/head
2025-12-04T12:26:06.7814570Z  * [new branch]                gh/guangyey/242/orig    -> origin/gh/guangyey/242/orig
2025-12-04T12:26:06.7814641Z  * [new branch]                gh/guangyey/243/base    -> origin/gh/guangyey/243/base
2025-12-04T12:26:06.7814714Z  * [new branch]                gh/guangyey/243/head    -> origin/gh/guangyey/243/head
2025-12-04T12:26:06.7814787Z  * [new branch]                gh/guangyey/243/orig    -> origin/gh/guangyey/243/orig
2025-12-04T12:26:06.7814857Z  * [new branch]                gh/guangyey/244/base    -> origin/gh/guangyey/244/base
2025-12-04T12:26:06.7814926Z  * [new branch]                gh/guangyey/244/head    -> origin/gh/guangyey/244/head
2025-12-04T12:26:06.7815000Z  * [new branch]                gh/guangyey/244/orig    -> origin/gh/guangyey/244/orig
2025-12-04T12:26:06.7815071Z  * [new branch]                gh/guangyey/245/base    -> origin/gh/guangyey/245/base
2025-12-04T12:26:06.7815143Z  * [new branch]                gh/guangyey/245/head    -> origin/gh/guangyey/245/head
2025-12-04T12:26:06.7815217Z  * [new branch]                gh/guangyey/245/orig    -> origin/gh/guangyey/245/orig
2025-12-04T12:26:06.7815288Z  * [new branch]                gh/guangyey/246/base    -> origin/gh/guangyey/246/base
2025-12-04T12:26:06.7815361Z  * [new branch]                gh/guangyey/246/head    -> origin/gh/guangyey/246/head
2025-12-04T12:26:06.7815434Z  * [new branch]                gh/guangyey/246/orig    -> origin/gh/guangyey/246/orig
2025-12-04T12:26:06.7815506Z  * [new branch]                gh/guangyey/247/base    -> origin/gh/guangyey/247/base
2025-12-04T12:26:06.7815578Z  * [new branch]                gh/guangyey/247/head    -> origin/gh/guangyey/247/head
2025-12-04T12:26:06.7815652Z  * [new branch]                gh/guangyey/247/orig    -> origin/gh/guangyey/247/orig
2025-12-04T12:26:06.7815722Z  * [new branch]                gh/guangyey/248/base    -> origin/gh/guangyey/248/base
2025-12-04T12:26:06.7815795Z  * [new branch]                gh/guangyey/248/head    -> origin/gh/guangyey/248/head
2025-12-04T12:26:06.7815866Z  * [new branch]                gh/guangyey/248/orig    -> origin/gh/guangyey/248/orig
2025-12-04T12:26:06.7815936Z  * [new branch]                gh/guangyey/249/base    -> origin/gh/guangyey/249/base
2025-12-04T12:26:06.7816008Z  * [new branch]                gh/guangyey/249/head    -> origin/gh/guangyey/249/head
2025-12-04T12:26:06.7816078Z  * [new branch]                gh/guangyey/249/orig    -> origin/gh/guangyey/249/orig
2025-12-04T12:26:06.7816148Z  * [new branch]                gh/guangyey/250/base    -> origin/gh/guangyey/250/base
2025-12-04T12:26:06.7816220Z  * [new branch]                gh/guangyey/250/head    -> origin/gh/guangyey/250/head
2025-12-04T12:26:06.7816293Z  * [new branch]                gh/guangyey/250/orig    -> origin/gh/guangyey/250/orig
2025-12-04T12:26:06.7816362Z  * [new branch]                gh/guangyey/251/base    -> origin/gh/guangyey/251/base
2025-12-04T12:26:06.7816462Z  * [new branch]                gh/guangyey/251/head    -> origin/gh/guangyey/251/head
2025-12-04T12:26:06.7816533Z  * [new branch]                gh/guangyey/251/orig    -> origin/gh/guangyey/251/orig
2025-12-04T12:26:06.7816603Z  * [new branch]                gh/guangyey/252/base    -> origin/gh/guangyey/252/base
2025-12-04T12:26:06.7816707Z  * [new branch]                gh/guangyey/252/head    -> origin/gh/guangyey/252/head
2025-12-04T12:26:06.7816818Z  * [new branch]                gh/guangyey/252/orig    -> origin/gh/guangyey/252/orig
2025-12-04T12:26:06.7816890Z  * [new branch]                gh/guangyey/253/base    -> origin/gh/guangyey/253/base
2025-12-04T12:26:06.7816962Z  * [new branch]                gh/guangyey/253/head    -> origin/gh/guangyey/253/head
2025-12-04T12:26:06.7817032Z  * [new branch]                gh/guangyey/253/orig    -> origin/gh/guangyey/253/orig
2025-12-04T12:26:06.7817102Z  * [new branch]                gh/guangyey/254/base    -> origin/gh/guangyey/254/base
2025-12-04T12:26:06.7817176Z  * [new branch]                gh/guangyey/254/head    -> origin/gh/guangyey/254/head
2025-12-04T12:26:06.7817246Z  * [new branch]                gh/guangyey/254/orig    -> origin/gh/guangyey/254/orig
2025-12-04T12:26:06.7817318Z  * [new branch]                gh/guangyey/255/base    -> origin/gh/guangyey/255/base
2025-12-04T12:26:06.7817389Z  * [new branch]                gh/guangyey/255/head    -> origin/gh/guangyey/255/head
2025-12-04T12:26:06.7817459Z  * [new branch]                gh/guangyey/255/orig    -> origin/gh/guangyey/255/orig
2025-12-04T12:26:06.7817530Z  * [new branch]                gh/guangyey/256/base    -> origin/gh/guangyey/256/base
2025-12-04T12:26:06.7817600Z  * [new branch]                gh/guangyey/256/head    -> origin/gh/guangyey/256/head
2025-12-04T12:26:06.7817669Z  * [new branch]                gh/guangyey/256/orig    -> origin/gh/guangyey/256/orig
2025-12-04T12:26:06.7817769Z  * [new branch]                gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base
2025-12-04T12:26:06.7817863Z  * [new branch]                gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head
2025-12-04T12:26:06.7817953Z  * [new branch]                gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig
2025-12-04T12:26:06.7818044Z  * [new branch]                gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base
2025-12-04T12:26:06.7818134Z  * [new branch]                gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head
2025-12-04T12:26:06.7818221Z  * [new branch]                gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig
2025-12-04T12:26:06.7818311Z  * [new branch]                gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base
2025-12-04T12:26:06.7818398Z  * [new branch]                gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head
2025-12-04T12:26:06.7818486Z  * [new branch]                gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig
2025-12-04T12:26:06.7818576Z  * [new branch]                gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base
2025-12-04T12:26:06.7818663Z  * [new branch]                gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head
2025-12-04T12:26:06.7818751Z  * [new branch]                gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig
2025-12-04T12:26:06.7818840Z  * [new branch]                gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base
2025-12-04T12:26:06.7818927Z  * [new branch]                gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head
2025-12-04T12:26:06.7819015Z  * [new branch]                gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig
2025-12-04T12:26:06.7819102Z  * [new branch]                gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base
2025-12-04T12:26:06.7819189Z  * [new branch]                gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head
2025-12-04T12:26:06.7819313Z  * [new branch]                gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig
2025-12-04T12:26:06.7819402Z  * [new branch]                gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base
2025-12-04T12:26:06.7819489Z  * [new branch]                gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head
2025-12-04T12:26:06.7819612Z  * [new branch]                gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig
2025-12-04T12:26:06.7819701Z  * [new branch]                gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base
2025-12-04T12:26:06.7819789Z  * [new branch]                gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head
2025-12-04T12:26:06.7819880Z  * [new branch]                gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig
2025-12-04T12:26:06.7819967Z  * [new branch]                gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base
2025-12-04T12:26:06.7820056Z  * [new branch]                gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head
2025-12-04T12:26:06.7820144Z  * [new branch]                gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig
2025-12-04T12:26:06.7820230Z  * [new branch]                gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base
2025-12-04T12:26:06.7820318Z  * [new branch]                gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head
2025-12-04T12:26:06.7820408Z  * [new branch]                gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig
2025-12-04T12:26:06.7820495Z  * [new branch]                gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base
2025-12-04T12:26:06.7820583Z  * [new branch]                gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head
2025-12-04T12:26:06.7820670Z  * [new branch]                gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig
2025-12-04T12:26:06.7820758Z  * [new branch]                gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base
2025-12-04T12:26:06.7820846Z  * [new branch]                gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head
2025-12-04T12:26:06.7820933Z  * [new branch]                gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig
2025-12-04T12:26:06.7821021Z  * [new branch]                gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base
2025-12-04T12:26:06.7821110Z  * [new branch]                gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head
2025-12-04T12:26:06.7821197Z  * [new branch]                gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig
2025-12-04T12:26:06.7821285Z  * [new branch]                gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base
2025-12-04T12:26:06.7821375Z  * [new branch]                gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head
2025-12-04T12:26:06.7821461Z  * [new branch]                gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig
2025-12-04T12:26:06.7821550Z  * [new branch]                gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base
2025-12-04T12:26:06.7821640Z  * [new branch]                gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head
2025-12-04T12:26:06.7821728Z  * [new branch]                gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig
2025-12-04T12:26:06.7821816Z  * [new branch]                gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base
2025-12-04T12:26:06.7821903Z  * [new branch]                gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head
2025-12-04T12:26:06.7821993Z  * [new branch]                gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig
2025-12-04T12:26:06.7822081Z  * [new branch]                gh/guilhermeleobas/253/base -> origin/gh/guilhermeleobas/253/base
2025-12-04T12:26:06.7822167Z  * [new branch]                gh/guilhermeleobas/253/head -> origin/gh/guilhermeleobas/253/head
2025-12-04T12:26:06.7822276Z  * [new branch]                gh/guilhermeleobas/253/orig -> origin/gh/guilhermeleobas/253/orig
2025-12-04T12:26:06.7822366Z  * [new branch]                gh/guilhermeleobas/254/base -> origin/gh/guilhermeleobas/254/base
2025-12-04T12:26:06.7822477Z  * [new branch]                gh/guilhermeleobas/254/head -> origin/gh/guilhermeleobas/254/head
2025-12-04T12:26:06.7822564Z  * [new branch]                gh/guilhermeleobas/254/orig -> origin/gh/guilhermeleobas/254/orig
2025-12-04T12:26:06.7822651Z  * [new branch]                gh/guilhermeleobas/255/base -> origin/gh/guilhermeleobas/255/base
2025-12-04T12:26:06.7822738Z  * [new branch]                gh/guilhermeleobas/255/head -> origin/gh/guilhermeleobas/255/head
2025-12-04T12:26:06.7822825Z  * [new branch]                gh/guilhermeleobas/255/orig -> origin/gh/guilhermeleobas/255/orig
2025-12-04T12:26:06.7822911Z  * [new branch]                gh/guilhermeleobas/256/base -> origin/gh/guilhermeleobas/256/base
2025-12-04T12:26:06.7823000Z  * [new branch]                gh/guilhermeleobas/256/head -> origin/gh/guilhermeleobas/256/head
2025-12-04T12:26:06.7823088Z  * [new branch]                gh/guilhermeleobas/256/orig -> origin/gh/guilhermeleobas/256/orig
2025-12-04T12:26:06.7823177Z  * [new branch]                gh/guilhermeleobas/257/base -> origin/gh/guilhermeleobas/257/base
2025-12-04T12:26:06.7823264Z  * [new branch]                gh/guilhermeleobas/257/head -> origin/gh/guilhermeleobas/257/head
2025-12-04T12:26:06.7823351Z  * [new branch]                gh/guilhermeleobas/257/orig -> origin/gh/guilhermeleobas/257/orig
2025-12-04T12:26:06.7823439Z  * [new branch]                gh/guilhermeleobas/258/base -> origin/gh/guilhermeleobas/258/base
2025-12-04T12:26:06.7823525Z  * [new branch]                gh/guilhermeleobas/258/head -> origin/gh/guilhermeleobas/258/head
2025-12-04T12:26:06.7823612Z  * [new branch]                gh/guilhermeleobas/258/orig -> origin/gh/guilhermeleobas/258/orig
2025-12-04T12:26:06.7823701Z  * [new branch]                gh/guilhermeleobas/259/base -> origin/gh/guilhermeleobas/259/base
2025-12-04T12:26:06.7823787Z  * [new branch]                gh/guilhermeleobas/259/head -> origin/gh/guilhermeleobas/259/head
2025-12-04T12:26:06.7823876Z  * [new branch]                gh/guilhermeleobas/259/orig -> origin/gh/guilhermeleobas/259/orig
2025-12-04T12:26:06.7823962Z  * [new branch]                gh/guilhermeleobas/260/base -> origin/gh/guilhermeleobas/260/base
2025-12-04T12:26:06.7824049Z  * [new branch]                gh/guilhermeleobas/260/head -> origin/gh/guilhermeleobas/260/head
2025-12-04T12:26:06.7824136Z  * [new branch]                gh/guilhermeleobas/260/orig -> origin/gh/guilhermeleobas/260/orig
2025-12-04T12:26:06.7824222Z  * [new branch]                gh/guilhermeleobas/261/base -> origin/gh/guilhermeleobas/261/base
2025-12-04T12:26:06.7824315Z  * [new branch]                gh/guilhermeleobas/261/head -> origin/gh/guilhermeleobas/261/head
2025-12-04T12:26:06.7824406Z  * [new branch]                gh/guilhermeleobas/261/orig -> origin/gh/guilhermeleobas/261/orig
2025-12-04T12:26:06.7824500Z  * [new branch]                gh/guilhermeleobas/262/base -> origin/gh/guilhermeleobas/262/base
2025-12-04T12:26:06.7824591Z  * [new branch]                gh/guilhermeleobas/262/head -> origin/gh/guilhermeleobas/262/head
2025-12-04T12:26:06.7824678Z  * [new branch]                gh/guilhermeleobas/262/orig -> origin/gh/guilhermeleobas/262/orig
2025-12-04T12:26:06.7824764Z  * [new branch]                gh/guilhermeleobas/263/base -> origin/gh/guilhermeleobas/263/base
2025-12-04T12:26:06.7824851Z  * [new branch]                gh/guilhermeleobas/263/head -> origin/gh/guilhermeleobas/263/head
2025-12-04T12:26:06.7824938Z  * [new branch]                gh/guilhermeleobas/263/orig -> origin/gh/guilhermeleobas/263/orig
2025-12-04T12:26:06.7825024Z  * [new branch]                gh/guilhermeleobas/264/base -> origin/gh/guilhermeleobas/264/base
2025-12-04T12:26:06.7825141Z  * [new branch]                gh/guilhermeleobas/264/head -> origin/gh/guilhermeleobas/264/head
2025-12-04T12:26:06.7825229Z  * [new branch]                gh/guilhermeleobas/264/orig -> origin/gh/guilhermeleobas/264/orig
2025-12-04T12:26:06.7825314Z  * [new branch]                gh/guilhermeleobas/265/base -> origin/gh/guilhermeleobas/265/base
2025-12-04T12:26:06.7825429Z  * [new branch]                gh/guilhermeleobas/265/head -> origin/gh/guilhermeleobas/265/head
2025-12-04T12:26:06.7825516Z  * [new branch]                gh/guilhermeleobas/265/orig -> origin/gh/guilhermeleobas/265/orig
2025-12-04T12:26:06.7825603Z  * [new branch]                gh/guilhermeleobas/266/base -> origin/gh/guilhermeleobas/266/base
2025-12-04T12:26:06.7825690Z  * [new branch]                gh/guilhermeleobas/266/head -> origin/gh/guilhermeleobas/266/head
2025-12-04T12:26:06.7825777Z  * [new branch]                gh/guilhermeleobas/266/orig -> origin/gh/guilhermeleobas/266/orig
2025-12-04T12:26:06.7825866Z  * [new branch]                gh/guilhermeleobas/267/base -> origin/gh/guilhermeleobas/267/base
2025-12-04T12:26:06.7825952Z  * [new branch]                gh/guilhermeleobas/267/head -> origin/gh/guilhermeleobas/267/head
2025-12-04T12:26:06.7826039Z  * [new branch]                gh/guilhermeleobas/267/orig -> origin/gh/guilhermeleobas/267/orig
2025-12-04T12:26:06.7826124Z  * [new branch]                gh/hameerabbasi/1/base  -> origin/gh/hameerabbasi/1/base
2025-12-04T12:26:06.7826202Z  * [new branch]                gh/hameerabbasi/1/head  -> origin/gh/hameerabbasi/1/head
2025-12-04T12:26:06.7826278Z  * [new branch]                gh/hameerabbasi/2/base  -> origin/gh/hameerabbasi/2/base
2025-12-04T12:26:06.7826354Z  * [new branch]                gh/hameerabbasi/2/head  -> origin/gh/hameerabbasi/2/head
2025-12-04T12:26:06.7826429Z  * [new branch]                gh/hameerabbasi/2/orig  -> origin/gh/hameerabbasi/2/orig
2025-12-04T12:26:06.7826502Z  * [new branch]                gh/hameerabbasi/3/base  -> origin/gh/hameerabbasi/3/base
2025-12-04T12:26:06.7826579Z  * [new branch]                gh/hameerabbasi/3/head  -> origin/gh/hameerabbasi/3/head
2025-12-04T12:26:06.7826653Z  * [new branch]                gh/hameerabbasi/3/orig  -> origin/gh/hameerabbasi/3/orig
2025-12-04T12:26:06.7826729Z  * [new branch]                gh/hameerabbasi/4/base  -> origin/gh/hameerabbasi/4/base
2025-12-04T12:26:06.7826849Z  * [new branch]                gh/hameerabbasi/4/head  -> origin/gh/hameerabbasi/4/head
2025-12-04T12:26:06.7826925Z  * [new branch]                gh/hameerabbasi/4/orig  -> origin/gh/hameerabbasi/4/orig
2025-12-04T12:26:06.7826995Z  * [new branch]                gh/huydhn/1/next        -> origin/gh/huydhn/1/next
2025-12-04T12:26:06.7827064Z  * [new branch]                gh/huydhn/2/next        -> origin/gh/huydhn/2/next
2025-12-04T12:26:06.7827130Z  * [new branch]                gh/huydhn/3/next        -> origin/gh/huydhn/3/next
2025-12-04T12:26:06.7827197Z  * [new branch]                gh/huydhn/4/next        -> origin/gh/huydhn/4/next
2025-12-04T12:26:06.7827263Z  * [new branch]                gh/huydhn/5/next        -> origin/gh/huydhn/5/next
2025-12-04T12:26:06.7827328Z  * [new branch]                gh/huydhn/6/next        -> origin/gh/huydhn/6/next
2025-12-04T12:26:06.7827397Z  * [new branch]                gh/int3/97/base         -> origin/gh/int3/97/base
2025-12-04T12:26:06.7827463Z  * [new branch]                gh/int3/97/head         -> origin/gh/int3/97/head
2025-12-04T12:26:06.7827534Z  * [new branch]                gh/isuruf/101/base      -> origin/gh/isuruf/101/base
2025-12-04T12:26:06.7827605Z  * [new branch]                gh/isuruf/101/head      -> origin/gh/isuruf/101/head
2025-12-04T12:26:06.7827674Z  * [new branch]                gh/isuruf/146/base      -> origin/gh/isuruf/146/base
2025-12-04T12:26:06.7827742Z  * [new branch]                gh/isuruf/146/head      -> origin/gh/isuruf/146/head
2025-12-04T12:26:06.7827810Z  * [new branch]                gh/isuruf/146/orig      -> origin/gh/isuruf/146/orig
2025-12-04T12:26:06.7827924Z  * [new branch]                gh/isuruf/158/base      -> origin/gh/isuruf/158/base
2025-12-04T12:26:06.7827992Z  * [new branch]                gh/isuruf/158/head      -> origin/gh/isuruf/158/head
2025-12-04T12:26:06.7828059Z  * [new branch]                gh/isuruf/159/base      -> origin/gh/isuruf/159/base
2025-12-04T12:26:06.7828167Z  * [new branch]                gh/isuruf/159/head      -> origin/gh/isuruf/159/head
2025-12-04T12:26:06.7828233Z  * [new branch]                gh/isuruf/160/base      -> origin/gh/isuruf/160/base
2025-12-04T12:26:06.7828300Z  * [new branch]                gh/isuruf/160/head      -> origin/gh/isuruf/160/head
2025-12-04T12:26:06.7828367Z  * [new branch]                gh/isuruf/160/orig      -> origin/gh/isuruf/160/orig
2025-12-04T12:26:06.7828435Z  * [new branch]                gh/isuruf/81/base       -> origin/gh/isuruf/81/base
2025-12-04T12:26:06.7828504Z  * [new branch]                gh/isuruf/81/head       -> origin/gh/isuruf/81/head
2025-12-04T12:26:06.7828574Z  * [new branch]                gh/isuruf/81/orig       -> origin/gh/isuruf/81/orig
2025-12-04T12:26:06.7828649Z  * [new branch]                gh/jamesjwu/176/base    -> origin/gh/jamesjwu/176/base
2025-12-04T12:26:06.7828725Z  * [new branch]                gh/jamesjwu/176/head    -> origin/gh/jamesjwu/176/head
2025-12-04T12:26:06.7828799Z  * [new branch]                gh/jamesjwu/176/orig    -> origin/gh/jamesjwu/176/orig
2025-12-04T12:26:06.7828873Z  * [new branch]                gh/jamesjwu/187/base    -> origin/gh/jamesjwu/187/base
2025-12-04T12:26:06.7828945Z  * [new branch]                gh/jamesjwu/187/head    -> origin/gh/jamesjwu/187/head
2025-12-04T12:26:06.7829016Z  * [new branch]                gh/jamesjwu/187/orig    -> origin/gh/jamesjwu/187/orig
2025-12-04T12:26:06.7829088Z  * [new branch]                gh/jamesjwu/196/base    -> origin/gh/jamesjwu/196/base
2025-12-04T12:26:06.7829159Z  * [new branch]                gh/jamesjwu/196/head    -> origin/gh/jamesjwu/196/head
2025-12-04T12:26:06.7829232Z  * [new branch]                gh/jamesjwu/196/orig    -> origin/gh/jamesjwu/196/orig
2025-12-04T12:26:06.7829305Z  * [new branch]                gh/jamesjwu/198/base    -> origin/gh/jamesjwu/198/base
2025-12-04T12:26:06.7829375Z  * [new branch]                gh/jamesjwu/198/head    -> origin/gh/jamesjwu/198/head
2025-12-04T12:26:06.7829447Z  * [new branch]                gh/jamesjwu/198/orig    -> origin/gh/jamesjwu/198/orig
2025-12-04T12:26:06.7829520Z  * [new branch]                gh/jamesjwu/207/base    -> origin/gh/jamesjwu/207/base
2025-12-04T12:26:06.7829591Z  * [new branch]                gh/jamesjwu/207/head    -> origin/gh/jamesjwu/207/head
2025-12-04T12:26:06.7829660Z  * [new branch]                gh/jamesjwu/207/orig    -> origin/gh/jamesjwu/207/orig
2025-12-04T12:26:06.7829733Z  * [new branch]                gh/jamesjwu/208/base    -> origin/gh/jamesjwu/208/base
2025-12-04T12:26:06.7829804Z  * [new branch]                gh/jamesjwu/208/head    -> origin/gh/jamesjwu/208/head
2025-12-04T12:26:06.7829875Z  * [new branch]                gh/jamesjwu/208/orig    -> origin/gh/jamesjwu/208/orig
2025-12-04T12:26:06.7829949Z  * [new branch]                gh/jamesjwu/52/base     -> origin/gh/jamesjwu/52/base
2025-12-04T12:26:06.7830022Z  * [new branch]                gh/jamesjwu/52/head     -> origin/gh/jamesjwu/52/head
2025-12-04T12:26:06.7830093Z  * [new branch]                gh/jamesjwu/53/base     -> origin/gh/jamesjwu/53/base
2025-12-04T12:26:06.7830168Z  * [new branch]                gh/jamesjwu/53/head     -> origin/gh/jamesjwu/53/head
2025-12-04T12:26:06.7830238Z  * [new branch]                gh/jamesjwu/54/base     -> origin/gh/jamesjwu/54/base
2025-12-04T12:26:06.7830308Z  * [new branch]                gh/jamesjwu/54/head     -> origin/gh/jamesjwu/54/head
2025-12-04T12:26:06.7830380Z  * [new branch]                gh/jamesjwu/55/base     -> origin/gh/jamesjwu/55/base
2025-12-04T12:26:06.7830449Z  * [new branch]                gh/jamesjwu/55/head     -> origin/gh/jamesjwu/55/head
2025-12-04T12:26:06.7830543Z  * [new branch]                gh/jamesjwu/56/base     -> origin/gh/jamesjwu/56/base
2025-12-04T12:26:06.7830615Z  * [new branch]                gh/jamesjwu/56/head     -> origin/gh/jamesjwu/56/head
2025-12-04T12:26:06.7830707Z  * [new branch]                gh/jamesjwu/57/base     -> origin/gh/jamesjwu/57/base
2025-12-04T12:26:06.7830777Z  * [new branch]                gh/jamesjwu/57/head     -> origin/gh/jamesjwu/57/head
2025-12-04T12:26:06.7830848Z  * [new branch]                gh/jamesjwu/58/base     -> origin/gh/jamesjwu/58/base
2025-12-04T12:26:06.7830917Z  * [new branch]                gh/jamesjwu/58/head     -> origin/gh/jamesjwu/58/head
2025-12-04T12:26:06.7830988Z  * [new branch]                gh/jamesjwu/59/base     -> origin/gh/jamesjwu/59/base
2025-12-04T12:26:06.7831058Z  * [new branch]                gh/jamesjwu/59/head     -> origin/gh/jamesjwu/59/head
2025-12-04T12:26:06.7831127Z  * [new branch]                gh/jamesjwu/60/base     -> origin/gh/jamesjwu/60/base
2025-12-04T12:26:06.7831200Z  * [new branch]                gh/jamesjwu/60/head     -> origin/gh/jamesjwu/60/head
2025-12-04T12:26:06.7831270Z  * [new branch]                gh/jamesjwu/61/base     -> origin/gh/jamesjwu/61/base
2025-12-04T12:26:06.7831341Z  * [new branch]                gh/jamesjwu/61/head     -> origin/gh/jamesjwu/61/head
2025-12-04T12:26:06.7831413Z  * [new branch]                gh/jamesjwu/62/base     -> origin/gh/jamesjwu/62/base
2025-12-04T12:26:06.7831483Z  * [new branch]                gh/jamesjwu/62/head     -> origin/gh/jamesjwu/62/head
2025-12-04T12:26:06.7831552Z  * [new branch]                gh/jamesjwu/63/base     -> origin/gh/jamesjwu/63/base
2025-12-04T12:26:06.7831624Z  * [new branch]                gh/jamesjwu/63/head     -> origin/gh/jamesjwu/63/head
2025-12-04T12:26:06.7831694Z  * [new branch]                gh/jamesjwu/64/base     -> origin/gh/jamesjwu/64/base
2025-12-04T12:26:06.7831764Z  * [new branch]                gh/jamesjwu/64/head     -> origin/gh/jamesjwu/64/head
2025-12-04T12:26:06.7831836Z  * [new branch]                gh/jamesjwu/65/base     -> origin/gh/jamesjwu/65/base
2025-12-04T12:26:06.7831905Z  * [new branch]                gh/jamesjwu/65/head     -> origin/gh/jamesjwu/65/head
2025-12-04T12:26:06.7831977Z  * [new branch]                gh/janeyx99/165/base    -> origin/gh/janeyx99/165/base
2025-12-04T12:26:06.7832051Z  * [new branch]                gh/janeyx99/165/head    -> origin/gh/janeyx99/165/head
2025-12-04T12:26:06.7832123Z  * [new branch]                gh/janeyx99/165/orig    -> origin/gh/janeyx99/165/orig
2025-12-04T12:26:06.7832196Z  * [new branch]                gh/janeyx99/201/base    -> origin/gh/janeyx99/201/base
2025-12-04T12:26:06.7832267Z  * [new branch]                gh/janeyx99/201/head    -> origin/gh/janeyx99/201/head
2025-12-04T12:26:06.7832339Z  * [new branch]                gh/janeyx99/201/orig    -> origin/gh/janeyx99/201/orig
2025-12-04T12:26:06.7832411Z  * [new branch]                gh/janeyx99/225/base    -> origin/gh/janeyx99/225/base
2025-12-04T12:26:06.7832483Z  * [new branch]                gh/janeyx99/225/head    -> origin/gh/janeyx99/225/head
2025-12-04T12:26:06.7832554Z  * [new branch]                gh/janeyx99/225/orig    -> origin/gh/janeyx99/225/orig
2025-12-04T12:26:06.7832628Z  * [new branch]                gh/janeyx99/299/base    -> origin/gh/janeyx99/299/base
2025-12-04T12:26:06.7832699Z  * [new branch]                gh/janeyx99/299/head    -> origin/gh/janeyx99/299/head
2025-12-04T12:26:06.7832768Z  * [new branch]                gh/janeyx99/299/orig    -> origin/gh/janeyx99/299/orig
2025-12-04T12:26:06.7832840Z  * [new branch]                gh/janeyx99/302/base    -> origin/gh/janeyx99/302/base
2025-12-04T12:26:06.7832911Z  * [new branch]                gh/janeyx99/302/head    -> origin/gh/janeyx99/302/head
2025-12-04T12:26:06.7832980Z  * [new branch]                gh/janeyx99/303/base    -> origin/gh/janeyx99/303/base
2025-12-04T12:26:06.7833052Z  * [new branch]                gh/janeyx99/303/head    -> origin/gh/janeyx99/303/head
2025-12-04T12:26:06.7833146Z  * [new branch]                gh/janeyx99/305/base    -> origin/gh/janeyx99/305/base
2025-12-04T12:26:06.7833217Z  * [new branch]                gh/janeyx99/305/head    -> origin/gh/janeyx99/305/head
2025-12-04T12:26:06.7833312Z  * [new branch]                gh/janeyx99/306/base    -> origin/gh/janeyx99/306/base
2025-12-04T12:26:06.7833381Z  * [new branch]                gh/janeyx99/306/head    -> origin/gh/janeyx99/306/head
2025-12-04T12:26:06.7833451Z  * [new branch]                gh/janeyx99/314/base    -> origin/gh/janeyx99/314/base
2025-12-04T12:26:06.7833522Z  * [new branch]                gh/janeyx99/314/head    -> origin/gh/janeyx99/314/head
2025-12-04T12:26:06.7833591Z  * [new branch]                gh/janeyx99/314/orig    -> origin/gh/janeyx99/314/orig
2025-12-04T12:26:06.7833663Z  * [new branch]                gh/janeyx99/315/base    -> origin/gh/janeyx99/315/base
2025-12-04T12:26:06.7833733Z  * [new branch]                gh/janeyx99/315/head    -> origin/gh/janeyx99/315/head
2025-12-04T12:26:06.7833804Z  * [new branch]                gh/janeyx99/315/orig    -> origin/gh/janeyx99/315/orig
2025-12-04T12:26:06.7833876Z  * [new branch]                gh/janeyx99/316/base    -> origin/gh/janeyx99/316/base
2025-12-04T12:26:06.7833947Z  * [new branch]                gh/janeyx99/316/head    -> origin/gh/janeyx99/316/head
2025-12-04T12:26:06.7834016Z  * [new branch]                gh/janeyx99/316/orig    -> origin/gh/janeyx99/316/orig
2025-12-04T12:26:06.7834086Z  * [new branch]                gh/janeyx99/317/base    -> origin/gh/janeyx99/317/base
2025-12-04T12:26:06.7834156Z  * [new branch]                gh/janeyx99/317/head    -> origin/gh/janeyx99/317/head
2025-12-04T12:26:06.7834224Z  * [new branch]                gh/janeyx99/317/orig    -> origin/gh/janeyx99/317/orig
2025-12-04T12:26:06.7834294Z  * [new branch]                gh/janeyx99/325/base    -> origin/gh/janeyx99/325/base
2025-12-04T12:26:06.7834367Z  * [new branch]                gh/janeyx99/325/head    -> origin/gh/janeyx99/325/head
2025-12-04T12:26:06.7834436Z  * [new branch]                gh/janeyx99/325/orig    -> origin/gh/janeyx99/325/orig
2025-12-04T12:26:06.7834506Z  * [new branch]                gh/janeyx99/327/base    -> origin/gh/janeyx99/327/base
2025-12-04T12:26:06.7834577Z  * [new branch]                gh/janeyx99/327/head    -> origin/gh/janeyx99/327/head
2025-12-04T12:26:06.7834646Z  * [new branch]                gh/janeyx99/327/orig    -> origin/gh/janeyx99/327/orig
2025-12-04T12:26:06.7834715Z  * [new branch]                gh/janeyx99/328/base    -> origin/gh/janeyx99/328/base
2025-12-04T12:26:06.7834784Z  * [new branch]                gh/janeyx99/328/head    -> origin/gh/janeyx99/328/head
2025-12-04T12:26:06.7834854Z  * [new branch]                gh/janeyx99/328/orig    -> origin/gh/janeyx99/328/orig
2025-12-04T12:26:06.7834924Z  * [new branch]                gh/janeyx99/329/base    -> origin/gh/janeyx99/329/base
2025-12-04T12:26:06.7834994Z  * [new branch]                gh/janeyx99/329/head    -> origin/gh/janeyx99/329/head
2025-12-04T12:26:06.7835064Z  * [new branch]                gh/janeyx99/329/orig    -> origin/gh/janeyx99/329/orig
2025-12-04T12:26:06.7835133Z  * [new branch]                gh/janeyx99/330/base    -> origin/gh/janeyx99/330/base
2025-12-04T12:26:06.7835203Z  * [new branch]                gh/janeyx99/330/head    -> origin/gh/janeyx99/330/head
2025-12-04T12:26:06.7835275Z  * [new branch]                gh/janeyx99/330/orig    -> origin/gh/janeyx99/330/orig
2025-12-04T12:26:06.7835344Z  * [new branch]                gh/janeyx99/331/base    -> origin/gh/janeyx99/331/base
2025-12-04T12:26:06.7835413Z  * [new branch]                gh/janeyx99/331/head    -> origin/gh/janeyx99/331/head
2025-12-04T12:26:06.7835484Z  * [new branch]                gh/janeyx99/331/orig    -> origin/gh/janeyx99/331/orig
2025-12-04T12:26:06.7835553Z  * [new branch]                gh/janeyx99/332/base    -> origin/gh/janeyx99/332/base
2025-12-04T12:26:06.7835643Z  * [new branch]                gh/janeyx99/332/head    -> origin/gh/janeyx99/332/head
2025-12-04T12:26:06.7835715Z  * [new branch]                gh/janeyx99/332/orig    -> origin/gh/janeyx99/332/orig
2025-12-04T12:26:06.7835784Z  * [new branch]                gh/janeyx99/333/base    -> origin/gh/janeyx99/333/base
2025-12-04T12:26:06.7835875Z  * [new branch]                gh/janeyx99/333/head    -> origin/gh/janeyx99/333/head
2025-12-04T12:26:06.7835946Z  * [new branch]                gh/janeyx99/333/orig    -> origin/gh/janeyx99/333/orig
2025-12-04T12:26:06.7836015Z  * [new branch]                gh/janeyx99/88/base     -> origin/gh/janeyx99/88/base
2025-12-04T12:26:06.7836085Z  * [new branch]                gh/janeyx99/88/head     -> origin/gh/janeyx99/88/head
2025-12-04T12:26:06.7836156Z  * [new branch]                gh/janeyx99/88/orig     -> origin/gh/janeyx99/88/orig
2025-12-04T12:26:06.7836226Z  * [new branch]                gh/jansel/360/base      -> origin/gh/jansel/360/base
2025-12-04T12:26:06.7836297Z  * [new branch]                gh/jansel/360/head      -> origin/gh/jansel/360/head
2025-12-04T12:26:06.7836368Z  * [new branch]                gh/jansel/451/base      -> origin/gh/jansel/451/base
2025-12-04T12:26:06.7836437Z  * [new branch]                gh/jansel/451/head      -> origin/gh/jansel/451/head
2025-12-04T12:26:06.7836505Z  * [new branch]                gh/jansel/451/orig      -> origin/gh/jansel/451/orig
2025-12-04T12:26:06.7836575Z  * [new branch]                gh/jansel/462/base      -> origin/gh/jansel/462/base
2025-12-04T12:26:06.7836642Z  * [new branch]                gh/jansel/462/head      -> origin/gh/jansel/462/head
2025-12-04T12:26:06.7836710Z  * [new branch]                gh/jansel/462/orig      -> origin/gh/jansel/462/orig
2025-12-04T12:26:06.7836821Z  * [new branch]                gh/jansel/533/base      -> origin/gh/jansel/533/base
2025-12-04T12:26:06.7836890Z  * [new branch]                gh/jansel/533/head      -> origin/gh/jansel/533/head
2025-12-04T12:26:06.7836959Z  * [new branch]                gh/jansel/533/orig      -> origin/gh/jansel/533/orig
2025-12-04T12:26:06.7837026Z  * [new branch]                gh/jansel/552/base      -> origin/gh/jansel/552/base
2025-12-04T12:26:06.7837093Z  * [new branch]                gh/jansel/552/head      -> origin/gh/jansel/552/head
2025-12-04T12:26:06.7837163Z  * [new branch]                gh/jansel/552/orig      -> origin/gh/jansel/552/orig
2025-12-04T12:26:06.7837230Z  * [new branch]                gh/jansel/553/base      -> origin/gh/jansel/553/base
2025-12-04T12:26:06.7837296Z  * [new branch]                gh/jansel/553/head      -> origin/gh/jansel/553/head
2025-12-04T12:26:06.7837364Z  * [new branch]                gh/jansel/553/orig      -> origin/gh/jansel/553/orig
2025-12-04T12:26:06.7837431Z  * [new branch]                gh/jansel/554/base      -> origin/gh/jansel/554/base
2025-12-04T12:26:06.7837499Z  * [new branch]                gh/jansel/554/head      -> origin/gh/jansel/554/head
2025-12-04T12:26:06.7837570Z  * [new branch]                gh/jansel/554/orig      -> origin/gh/jansel/554/orig
2025-12-04T12:26:06.7837638Z  * [new branch]                gh/jansel/555/base      -> origin/gh/jansel/555/base
2025-12-04T12:26:06.7837705Z  * [new branch]                gh/jansel/555/head      -> origin/gh/jansel/555/head
2025-12-04T12:26:06.7837775Z  * [new branch]                gh/jansel/555/orig      -> origin/gh/jansel/555/orig
2025-12-04T12:26:06.7837841Z  * [new branch]                gh/jansel/556/base      -> origin/gh/jansel/556/base
2025-12-04T12:26:06.7837909Z  * [new branch]                gh/jansel/556/head      -> origin/gh/jansel/556/head
2025-12-04T12:26:06.7837979Z  * [new branch]                gh/jansel/556/orig      -> origin/gh/jansel/556/orig
2025-12-04T12:26:06.7838046Z  * [new branch]                gh/jansel/557/base      -> origin/gh/jansel/557/base
2025-12-04T12:26:06.7838113Z  * [new branch]                gh/jansel/557/head      -> origin/gh/jansel/557/head
2025-12-04T12:26:06.7838233Z  * [new branch]                gh/jansel/557/orig      -> origin/gh/jansel/557/orig
2025-12-04T12:26:06.7838302Z  * [new branch]                gh/jansel/558/base      -> origin/gh/jansel/558/base
2025-12-04T12:26:06.7838372Z  * [new branch]                gh/jansel/558/head      -> origin/gh/jansel/558/head
2025-12-04T12:26:06.7838479Z  * [new branch]                gh/jansel/558/orig      -> origin/gh/jansel/558/orig
2025-12-04T12:26:06.7838547Z  * [new branch]                gh/jansel/559/base      -> origin/gh/jansel/559/base
2025-12-04T12:26:06.7838616Z  * [new branch]                gh/jansel/559/head      -> origin/gh/jansel/559/head
2025-12-04T12:26:06.7838684Z  * [new branch]                gh/jansel/559/orig      -> origin/gh/jansel/559/orig
2025-12-04T12:26:06.7838751Z  * [new branch]                gh/jansel/560/base      -> origin/gh/jansel/560/base
2025-12-04T12:26:06.7838820Z  * [new branch]                gh/jansel/560/head      -> origin/gh/jansel/560/head
2025-12-04T12:26:06.7838887Z  * [new branch]                gh/jansel/560/orig      -> origin/gh/jansel/560/orig
2025-12-04T12:26:06.7838956Z  * [new branch]                gh/jansel/561/base      -> origin/gh/jansel/561/base
2025-12-04T12:26:06.7839025Z  * [new branch]                gh/jansel/561/head      -> origin/gh/jansel/561/head
2025-12-04T12:26:06.7839095Z  * [new branch]                gh/jansel/561/orig      -> origin/gh/jansel/561/orig
2025-12-04T12:26:06.7839162Z  * [new branch]                gh/jansel/562/base      -> origin/gh/jansel/562/base
2025-12-04T12:26:06.7839232Z  * [new branch]                gh/jansel/562/head      -> origin/gh/jansel/562/head
2025-12-04T12:26:06.7839300Z  * [new branch]                gh/jansel/562/orig      -> origin/gh/jansel/562/orig
2025-12-04T12:26:06.7839370Z  * [new branch]                gh/jansel/563/base      -> origin/gh/jansel/563/base
2025-12-04T12:26:06.7839439Z  * [new branch]                gh/jansel/563/head      -> origin/gh/jansel/563/head
2025-12-04T12:26:06.7839505Z  * [new branch]                gh/jansel/563/orig      -> origin/gh/jansel/563/orig
2025-12-04T12:26:06.7839575Z  * [new branch]                gh/jansel/564/base      -> origin/gh/jansel/564/base
2025-12-04T12:26:06.7839645Z  * [new branch]                gh/jansel/564/head      -> origin/gh/jansel/564/head
2025-12-04T12:26:06.7839713Z  * [new branch]                gh/jansel/564/orig      -> origin/gh/jansel/564/orig
2025-12-04T12:26:06.7839781Z  * [new branch]                gh/jansel/565/base      -> origin/gh/jansel/565/base
2025-12-04T12:26:06.7839850Z  * [new branch]                gh/jansel/565/head      -> origin/gh/jansel/565/head
2025-12-04T12:26:06.7839916Z  * [new branch]                gh/jansel/565/orig      -> origin/gh/jansel/565/orig
2025-12-04T12:26:06.7839983Z  * [new branch]                gh/jansel/566/base      -> origin/gh/jansel/566/base
2025-12-04T12:26:06.7840051Z  * [new branch]                gh/jansel/566/head      -> origin/gh/jansel/566/head
2025-12-04T12:26:06.7840118Z  * [new branch]                gh/jansel/566/orig      -> origin/gh/jansel/566/orig
2025-12-04T12:26:06.7840188Z  * [new branch]                gh/jansel/567/base      -> origin/gh/jansel/567/base
2025-12-04T12:26:06.7840255Z  * [new branch]                gh/jansel/567/head      -> origin/gh/jansel/567/head
2025-12-04T12:26:06.7840323Z  * [new branch]                gh/jansel/567/orig      -> origin/gh/jansel/567/orig
2025-12-04T12:26:06.7840392Z  * [new branch]                gh/jansel/568/base      -> origin/gh/jansel/568/base
2025-12-04T12:26:06.7840459Z  * [new branch]                gh/jansel/568/head      -> origin/gh/jansel/568/head
2025-12-04T12:26:06.7840526Z  * [new branch]                gh/jansel/568/orig      -> origin/gh/jansel/568/orig
2025-12-04T12:26:06.7840594Z  * [new branch]                gh/jansel/569/base      -> origin/gh/jansel/569/base
2025-12-04T12:26:06.7840660Z  * [new branch]                gh/jansel/569/head      -> origin/gh/jansel/569/head
2025-12-04T12:26:06.7840726Z  * [new branch]                gh/jansel/569/orig      -> origin/gh/jansel/569/orig
2025-12-04T12:26:06.7840817Z  * [new branch]                gh/jansel/570/base      -> origin/gh/jansel/570/base
2025-12-04T12:26:06.7840886Z  * [new branch]                gh/jansel/570/head      -> origin/gh/jansel/570/head
2025-12-04T12:26:06.7840978Z  * [new branch]                gh/jansel/570/orig      -> origin/gh/jansel/570/orig
2025-12-04T12:26:06.7841047Z  * [new branch]                gh/jansel/571/base      -> origin/gh/jansel/571/base
2025-12-04T12:26:06.7841114Z  * [new branch]                gh/jansel/571/head      -> origin/gh/jansel/571/head
2025-12-04T12:26:06.7841180Z  * [new branch]                gh/jansel/571/orig      -> origin/gh/jansel/571/orig
2025-12-04T12:26:06.7841248Z  * [new branch]                gh/jansel/572/base      -> origin/gh/jansel/572/base
2025-12-04T12:26:06.7841315Z  * [new branch]                gh/jansel/572/head      -> origin/gh/jansel/572/head
2025-12-04T12:26:06.7841381Z  * [new branch]                gh/jansel/572/orig      -> origin/gh/jansel/572/orig
2025-12-04T12:26:06.7841450Z  * [new branch]                gh/jansel/573/base      -> origin/gh/jansel/573/base
2025-12-04T12:26:06.7841517Z  * [new branch]                gh/jansel/573/head      -> origin/gh/jansel/573/head
2025-12-04T12:26:06.7841588Z  * [new branch]                gh/jansel/573/orig      -> origin/gh/jansel/573/orig
2025-12-04T12:26:06.7841656Z  * [new branch]                gh/jansel/574/base      -> origin/gh/jansel/574/base
2025-12-04T12:26:06.7841723Z  * [new branch]                gh/jansel/574/head      -> origin/gh/jansel/574/head
2025-12-04T12:26:06.7841792Z  * [new branch]                gh/jansel/574/orig      -> origin/gh/jansel/574/orig
2025-12-04T12:26:06.7841859Z  * [new branch]                gh/jansel/575/base      -> origin/gh/jansel/575/base
2025-12-04T12:26:06.7841925Z  * [new branch]                gh/jansel/575/head      -> origin/gh/jansel/575/head
2025-12-04T12:26:06.7841993Z  * [new branch]                gh/jansel/575/orig      -> origin/gh/jansel/575/orig
2025-12-04T12:26:06.7842062Z  * [new branch]                gh/jansel/576/base      -> origin/gh/jansel/576/base
2025-12-04T12:26:06.7842129Z  * [new branch]                gh/jansel/576/head      -> origin/gh/jansel/576/head
2025-12-04T12:26:06.7842197Z  * [new branch]                gh/jansel/576/orig      -> origin/gh/jansel/576/orig
2025-12-04T12:26:06.7842278Z  * [new branch]                gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base
2025-12-04T12:26:06.7842357Z  * [new branch]                gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head
2025-12-04T12:26:06.7842434Z  * [new branch]                gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig
2025-12-04T12:26:06.7842509Z  * [new branch]                gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base
2025-12-04T12:26:06.7842583Z  * [new branch]                gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head
2025-12-04T12:26:06.7842659Z  * [new branch]                gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig
2025-12-04T12:26:06.7842736Z  * [new branch]                gh/jerryzh168/1/base    -> origin/gh/jerryzh168/1/base
2025-12-04T12:26:06.7842809Z  * [new branch]                gh/jerryzh168/1/head    -> origin/gh/jerryzh168/1/head
2025-12-04T12:26:06.7842882Z  * [new branch]                gh/jerryzh168/1/orig    -> origin/gh/jerryzh168/1/orig
2025-12-04T12:26:06.7842954Z  * [new branch]                gh/jiayisunx/59/base    -> origin/gh/jiayisunx/59/base
2025-12-04T12:26:06.7843025Z  * [new branch]                gh/jiayisunx/59/head    -> origin/gh/jiayisunx/59/head
2025-12-04T12:26:06.7843097Z  * [new branch]                gh/jiayisunx/59/orig    -> origin/gh/jiayisunx/59/orig
2025-12-04T12:26:06.7843167Z  * [new branch]                gh/jiayisunx/61/base    -> origin/gh/jiayisunx/61/base
2025-12-04T12:26:06.7843238Z  * [new branch]                gh/jiayisunx/61/head    -> origin/gh/jiayisunx/61/head
2025-12-04T12:26:06.7843310Z  * [new branch]                gh/jiayisunx/61/orig    -> origin/gh/jiayisunx/61/orig
2025-12-04T12:26:06.7843410Z  * [new branch]                gh/jiayisunx/68/base    -> origin/gh/jiayisunx/68/base
2025-12-04T12:26:06.7843483Z  * [new branch]                gh/jiayisunx/68/head    -> origin/gh/jiayisunx/68/head
2025-12-04T12:26:06.7843581Z  * [new branch]                gh/jiayisunx/68/orig    -> origin/gh/jiayisunx/68/orig
2025-12-04T12:26:06.7843652Z  * [new branch]                gh/jiayisunx/77/base    -> origin/gh/jiayisunx/77/base
2025-12-04T12:26:06.7843724Z  * [new branch]                gh/jiayisunx/77/head    -> origin/gh/jiayisunx/77/head
2025-12-04T12:26:06.7843794Z  * [new branch]                gh/jiayisunx/77/orig    -> origin/gh/jiayisunx/77/orig
2025-12-04T12:26:06.7843865Z  * [new branch]                gh/jiayisunx/78/base    -> origin/gh/jiayisunx/78/base
2025-12-04T12:26:06.7843939Z  * [new branch]                gh/jiayisunx/78/head    -> origin/gh/jiayisunx/78/head
2025-12-04T12:26:06.7844010Z  * [new branch]                gh/jiayisunx/78/orig    -> origin/gh/jiayisunx/78/orig
2025-12-04T12:26:06.7844081Z  * [new branch]                gh/jiayisunx/79/base    -> origin/gh/jiayisunx/79/base
2025-12-04T12:26:06.7844152Z  * [new branch]                gh/jiayisunx/79/head    -> origin/gh/jiayisunx/79/head
2025-12-04T12:26:06.7844224Z  * [new branch]                gh/jiayisunx/79/orig    -> origin/gh/jiayisunx/79/orig
2025-12-04T12:26:06.7844294Z  * [new branch]                gh/jiayisunx/82/base    -> origin/gh/jiayisunx/82/base
2025-12-04T12:26:06.7844366Z  * [new branch]                gh/jiayisunx/82/head    -> origin/gh/jiayisunx/82/head
2025-12-04T12:26:06.7844437Z  * [new branch]                gh/jiayisunx/82/orig    -> origin/gh/jiayisunx/82/orig
2025-12-04T12:26:06.7844507Z  * [new branch]                gh/jiayisunx/83/base    -> origin/gh/jiayisunx/83/base
2025-12-04T12:26:06.7844580Z  * [new branch]                gh/jiayisunx/83/head    -> origin/gh/jiayisunx/83/head
2025-12-04T12:26:06.7844652Z  * [new branch]                gh/jiayisunx/83/orig    -> origin/gh/jiayisunx/83/orig
2025-12-04T12:26:06.7844723Z  * [new branch]                gh/jiayisunx/84/base    -> origin/gh/jiayisunx/84/base
2025-12-04T12:26:06.7844795Z  * [new branch]                gh/jiayisunx/84/head    -> origin/gh/jiayisunx/84/head
2025-12-04T12:26:06.7844868Z  * [new branch]                gh/jiayisunx/84/orig    -> origin/gh/jiayisunx/84/orig
2025-12-04T12:26:06.7844939Z  * [new branch]                gh/jiayisunx/85/base    -> origin/gh/jiayisunx/85/base
2025-12-04T12:26:06.7845010Z  * [new branch]                gh/jiayisunx/85/head    -> origin/gh/jiayisunx/85/head
2025-12-04T12:26:06.7845082Z  * [new branch]                gh/jiayisunx/85/orig    -> origin/gh/jiayisunx/85/orig
2025-12-04T12:26:06.7845154Z  * [new branch]                gh/jiayisunx/86/base    -> origin/gh/jiayisunx/86/base
2025-12-04T12:26:06.7845224Z  * [new branch]                gh/jiayisunx/86/head    -> origin/gh/jiayisunx/86/head
2025-12-04T12:26:06.7845296Z  * [new branch]                gh/jiayisunx/86/orig    -> origin/gh/jiayisunx/86/orig
2025-12-04T12:26:06.7845367Z  * [new branch]                gh/jiayisunx/87/base    -> origin/gh/jiayisunx/87/base
2025-12-04T12:26:06.7845438Z  * [new branch]                gh/jiayisunx/87/head    -> origin/gh/jiayisunx/87/head
2025-12-04T12:26:06.7845509Z  * [new branch]                gh/jiayisunx/87/orig    -> origin/gh/jiayisunx/87/orig
2025-12-04T12:26:06.7845581Z  * [new branch]                gh/jiayisunx/88/base    -> origin/gh/jiayisunx/88/base
2025-12-04T12:26:06.7845650Z  * [new branch]                gh/jiayisunx/88/head    -> origin/gh/jiayisunx/88/head
2025-12-04T12:26:06.7845720Z  * [new branch]                gh/jiayisunx/88/orig    -> origin/gh/jiayisunx/88/orig
2025-12-04T12:26:06.7845792Z  * [new branch]                gh/jiayisunx/89/base    -> origin/gh/jiayisunx/89/base
2025-12-04T12:26:06.7845862Z  * [new branch]                gh/jiayisunx/89/head    -> origin/gh/jiayisunx/89/head
2025-12-04T12:26:06.7845958Z  * [new branch]                gh/jiayisunx/89/orig    -> origin/gh/jiayisunx/89/orig
2025-12-04T12:26:06.7846032Z  * [new branch]                gh/jiayisunx/90/base    -> origin/gh/jiayisunx/90/base
2025-12-04T12:26:06.7846103Z  * [new branch]                gh/jiayisunx/90/head    -> origin/gh/jiayisunx/90/head
2025-12-04T12:26:06.7846213Z  * [new branch]                gh/jiayisunx/90/orig    -> origin/gh/jiayisunx/90/orig
2025-12-04T12:26:06.7846295Z  * [new branch]                gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base
2025-12-04T12:26:06.7846369Z  * [new branch]                gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head
2025-12-04T12:26:06.7846439Z  * [new branch]                gh/jturney/1/base       -> origin/gh/jturney/1/base
2025-12-04T12:26:06.7846507Z  * [new branch]                gh/jturney/1/head       -> origin/gh/jturney/1/head
2025-12-04T12:26:06.7846574Z  * [new branch]                gh/jturney/1/orig       -> origin/gh/jturney/1/orig
2025-12-04T12:26:06.7846643Z  * [new branch]                gh/jturney/2/base       -> origin/gh/jturney/2/base
2025-12-04T12:26:06.7846710Z  * [new branch]                gh/jturney/2/head       -> origin/gh/jturney/2/head
2025-12-04T12:26:06.7846825Z  * [new branch]                gh/jturney/2/orig       -> origin/gh/jturney/2/orig
2025-12-04T12:26:06.7846907Z  * [new branch]                gh/karthickai/10/base   -> origin/gh/karthickai/10/base
2025-12-04T12:26:06.7846983Z  * [new branch]                gh/karthickai/10/head   -> origin/gh/karthickai/10/head
2025-12-04T12:26:06.7847056Z  * [new branch]                gh/karthickai/10/orig   -> origin/gh/karthickai/10/orig
2025-12-04T12:26:06.7847129Z  * [new branch]                gh/karthickai/11/base   -> origin/gh/karthickai/11/base
2025-12-04T12:26:06.7847203Z  * [new branch]                gh/karthickai/11/head   -> origin/gh/karthickai/11/head
2025-12-04T12:26:06.7847276Z  * [new branch]                gh/karthickai/11/orig   -> origin/gh/karthickai/11/orig
2025-12-04T12:26:06.7847352Z  * [new branch]                gh/karthickai/12/base   -> origin/gh/karthickai/12/base
2025-12-04T12:26:06.7847425Z  * [new branch]                gh/karthickai/12/head   -> origin/gh/karthickai/12/head
2025-12-04T12:26:06.7847502Z  * [new branch]                gh/karthickai/12/orig   -> origin/gh/karthickai/12/orig
2025-12-04T12:26:06.7847575Z  * [new branch]                gh/karthickai/13/base   -> origin/gh/karthickai/13/base
2025-12-04T12:26:06.7847648Z  * [new branch]                gh/karthickai/13/head   -> origin/gh/karthickai/13/head
2025-12-04T12:26:06.7847720Z  * [new branch]                gh/karthickai/13/orig   -> origin/gh/karthickai/13/orig
2025-12-04T12:26:06.7847794Z  * [new branch]                gh/karthickai/14/base   -> origin/gh/karthickai/14/base
2025-12-04T12:26:06.7847866Z  * [new branch]                gh/karthickai/14/head   -> origin/gh/karthickai/14/head
2025-12-04T12:26:06.7847938Z  * [new branch]                gh/karthickai/14/orig   -> origin/gh/karthickai/14/orig
2025-12-04T12:26:06.7848012Z  * [new branch]                gh/karthickai/15/base   -> origin/gh/karthickai/15/base
2025-12-04T12:26:06.7848085Z  * [new branch]                gh/karthickai/15/head   -> origin/gh/karthickai/15/head
2025-12-04T12:26:06.7848160Z  * [new branch]                gh/karthickai/15/orig   -> origin/gh/karthickai/15/orig
2025-12-04T12:26:06.7848232Z  * [new branch]                gh/karthickai/16/base   -> origin/gh/karthickai/16/base
2025-12-04T12:26:06.7848305Z  * [new branch]                gh/karthickai/16/head   -> origin/gh/karthickai/16/head
2025-12-04T12:26:06.7848378Z  * [new branch]                gh/karthickai/16/orig   -> origin/gh/karthickai/16/orig
2025-12-04T12:26:06.7848451Z  * [new branch]                gh/karthickai/17/base   -> origin/gh/karthickai/17/base
2025-12-04T12:26:06.7848523Z  * [new branch]                gh/karthickai/17/head   -> origin/gh/karthickai/17/head
2025-12-04T12:26:06.7848596Z  * [new branch]                gh/karthickai/17/orig   -> origin/gh/karthickai/17/orig
2025-12-04T12:26:06.7848700Z  * [new branch]                gh/karthickai/18/base   -> origin/gh/karthickai/18/base
2025-12-04T12:26:06.7848774Z  * [new branch]                gh/karthickai/18/head   -> origin/gh/karthickai/18/head
2025-12-04T12:26:06.7848883Z  * [new branch]                gh/karthickai/18/orig   -> origin/gh/karthickai/18/orig
2025-12-04T12:26:06.7848956Z  * [new branch]                gh/karthickai/19/base   -> origin/gh/karthickai/19/base
2025-12-04T12:26:06.7849028Z  * [new branch]                gh/karthickai/19/head   -> origin/gh/karthickai/19/head
2025-12-04T12:26:06.7849102Z  * [new branch]                gh/karthickai/19/orig   -> origin/gh/karthickai/19/orig
2025-12-04T12:26:06.7849174Z  * [new branch]                gh/karthickai/20/base   -> origin/gh/karthickai/20/base
2025-12-04T12:26:06.7849245Z  * [new branch]                gh/karthickai/20/head   -> origin/gh/karthickai/20/head
2025-12-04T12:26:06.7849319Z  * [new branch]                gh/karthickai/20/orig   -> origin/gh/karthickai/20/orig
2025-12-04T12:26:06.7849391Z  * [new branch]                gh/karthickai/21/base   -> origin/gh/karthickai/21/base
2025-12-04T12:26:06.7849463Z  * [new branch]                gh/karthickai/21/head   -> origin/gh/karthickai/21/head
2025-12-04T12:26:06.7849538Z  * [new branch]                gh/karthickai/21/orig   -> origin/gh/karthickai/21/orig
2025-12-04T12:26:06.7849610Z  * [new branch]                gh/karthickai/22/base   -> origin/gh/karthickai/22/base
2025-12-04T12:26:06.7849683Z  * [new branch]                gh/karthickai/22/head   -> origin/gh/karthickai/22/head
2025-12-04T12:26:06.7849756Z  * [new branch]                gh/karthickai/22/orig   -> origin/gh/karthickai/22/orig
2025-12-04T12:26:06.7849828Z  * [new branch]                gh/karthickai/23/base   -> origin/gh/karthickai/23/base
2025-12-04T12:26:06.7849902Z  * [new branch]                gh/karthickai/23/head   -> origin/gh/karthickai/23/head
2025-12-04T12:26:06.7849974Z  * [new branch]                gh/karthickai/23/orig   -> origin/gh/karthickai/23/orig
2025-12-04T12:26:06.7850046Z  * [new branch]                gh/karthickai/24/base   -> origin/gh/karthickai/24/base
2025-12-04T12:26:06.7850118Z  * [new branch]                gh/karthickai/24/head   -> origin/gh/karthickai/24/head
2025-12-04T12:26:06.7850192Z  * [new branch]                gh/karthickai/24/orig   -> origin/gh/karthickai/24/orig
2025-12-04T12:26:06.7850264Z  * [new branch]                gh/karthickai/25/base   -> origin/gh/karthickai/25/base
2025-12-04T12:26:06.7850336Z  * [new branch]                gh/karthickai/25/head   -> origin/gh/karthickai/25/head
2025-12-04T12:26:06.7850408Z  * [new branch]                gh/karthickai/25/orig   -> origin/gh/karthickai/25/orig
2025-12-04T12:26:06.7850484Z  * [new branch]                gh/karthickai/26/base   -> origin/gh/karthickai/26/base
2025-12-04T12:26:06.7850560Z  * [new branch]                gh/karthickai/26/head   -> origin/gh/karthickai/26/head
2025-12-04T12:26:06.7850634Z  * [new branch]                gh/karthickai/26/orig   -> origin/gh/karthickai/26/orig
2025-12-04T12:26:06.7850708Z  * [new branch]                gh/karthickai/6/base    -> origin/gh/karthickai/6/base
2025-12-04T12:26:06.7850784Z  * [new branch]                gh/karthickai/6/head    -> origin/gh/karthickai/6/head
2025-12-04T12:26:06.7850857Z  * [new branch]                gh/karthickai/6/orig    -> origin/gh/karthickai/6/orig
2025-12-04T12:26:06.7850924Z  * [new branch]                gh/krocki/1/base        -> origin/gh/krocki/1/base
2025-12-04T12:26:06.7850995Z  * [new branch]                gh/krocki/1/head        -> origin/gh/krocki/1/head
2025-12-04T12:26:06.7851063Z  * [new branch]                gh/krocki/1/orig        -> origin/gh/krocki/1/orig
2025-12-04T12:26:06.7851336Z  * [new branch]                gh/krocki/2/base        -> origin/gh/krocki/2/base
2025-12-04T12:26:06.7851403Z  * [new branch]                gh/krocki/2/head        -> origin/gh/krocki/2/head
2025-12-04T12:26:06.7851500Z  * [new branch]                gh/krocki/2/orig        -> origin/gh/krocki/2/orig
2025-12-04T12:26:06.7851585Z  * [new branch]                gh/kurtamohler/60/base  -> origin/gh/kurtamohler/60/base
2025-12-04T12:26:06.7851663Z  * [new branch]                gh/kurtamohler/60/head  -> origin/gh/kurtamohler/60/head
2025-12-04T12:26:06.7851765Z  * [new branch]                gh/kurtamohler/60/orig  -> origin/gh/kurtamohler/60/orig
2025-12-04T12:26:06.7851843Z  * [new branch]                gh/kurtamohler/61/base  -> origin/gh/kurtamohler/61/base
2025-12-04T12:26:06.7851918Z  * [new branch]                gh/kurtamohler/61/head  -> origin/gh/kurtamohler/61/head
2025-12-04T12:26:06.7851992Z  * [new branch]                gh/kurtamohler/61/orig  -> origin/gh/kurtamohler/61/orig
2025-12-04T12:26:06.7852070Z  * [new branch]                gh/kurtamohler/62/base  -> origin/gh/kurtamohler/62/base
2025-12-04T12:26:06.7852145Z  * [new branch]                gh/kurtamohler/62/head  -> origin/gh/kurtamohler/62/head
2025-12-04T12:26:06.7852223Z  * [new branch]                gh/kurtamohler/62/orig  -> origin/gh/kurtamohler/62/orig
2025-12-04T12:26:06.7852301Z  * [new branch]                gh/kurtamohler/63/base  -> origin/gh/kurtamohler/63/base
2025-12-04T12:26:06.7852379Z  * [new branch]                gh/kurtamohler/63/head  -> origin/gh/kurtamohler/63/head
2025-12-04T12:26:06.7852454Z  * [new branch]                gh/kurtamohler/63/orig  -> origin/gh/kurtamohler/63/orig
2025-12-04T12:26:06.7852534Z  * [new branch]                gh/kurtamohler/64/base  -> origin/gh/kurtamohler/64/base
2025-12-04T12:26:06.7852610Z  * [new branch]                gh/kurtamohler/64/head  -> origin/gh/kurtamohler/64/head
2025-12-04T12:26:06.7852685Z  * [new branch]                gh/kurtamohler/64/orig  -> origin/gh/kurtamohler/64/orig
2025-12-04T12:26:06.7852764Z  * [new branch]                gh/kurtamohler/65/base  -> origin/gh/kurtamohler/65/base
2025-12-04T12:26:06.7852837Z  * [new branch]                gh/kurtamohler/65/head  -> origin/gh/kurtamohler/65/head
2025-12-04T12:26:06.7852917Z  * [new branch]                gh/kurtamohler/65/orig  -> origin/gh/kurtamohler/65/orig
2025-12-04T12:26:06.7852992Z  * [new branch]                gh/kurtamohler/66/base  -> origin/gh/kurtamohler/66/base
2025-12-04T12:26:06.7853069Z  * [new branch]                gh/kurtamohler/66/head  -> origin/gh/kurtamohler/66/head
2025-12-04T12:26:06.7853145Z  * [new branch]                gh/kurtamohler/66/orig  -> origin/gh/kurtamohler/66/orig
2025-12-04T12:26:06.7853220Z  * [new branch]                gh/kurtamohler/67/base  -> origin/gh/kurtamohler/67/base
2025-12-04T12:26:06.7853294Z  * [new branch]                gh/kurtamohler/67/head  -> origin/gh/kurtamohler/67/head
2025-12-04T12:26:06.7853371Z  * [new branch]                gh/kurtamohler/67/orig  -> origin/gh/kurtamohler/67/orig
2025-12-04T12:26:06.7853444Z  * [new branch]                gh/kwen2501/130/base    -> origin/gh/kwen2501/130/base
2025-12-04T12:26:06.7853515Z  * [new branch]                gh/kwen2501/130/head    -> origin/gh/kwen2501/130/head
2025-12-04T12:26:06.7853591Z  * [new branch]                gh/kwen2501/130/orig    -> origin/gh/kwen2501/130/orig
2025-12-04T12:26:06.7853663Z  * [new branch]                gh/kwen2501/170/base    -> origin/gh/kwen2501/170/base
2025-12-04T12:26:06.7853737Z  * [new branch]                gh/kwen2501/170/head    -> origin/gh/kwen2501/170/head
2025-12-04T12:26:06.7853808Z  * [new branch]                gh/kwen2501/187/base    -> origin/gh/kwen2501/187/base
2025-12-04T12:26:06.7853877Z  * [new branch]                gh/kwen2501/187/head    -> origin/gh/kwen2501/187/head
2025-12-04T12:26:06.7853947Z  * [new branch]                gh/kwen2501/187/orig    -> origin/gh/kwen2501/187/orig
2025-12-04T12:26:06.7854017Z  * [new branch]                gh/kwen2501/188/base    -> origin/gh/kwen2501/188/base
2025-12-04T12:26:06.7854086Z  * [new branch]                gh/kwen2501/188/head    -> origin/gh/kwen2501/188/head
2025-12-04T12:26:06.7854179Z  * [new branch]                gh/kwen2501/188/orig    -> origin/gh/kwen2501/188/orig
2025-12-04T12:26:06.7854252Z  * [new branch]                gh/kwen2501/211/base    -> origin/gh/kwen2501/211/base
2025-12-04T12:26:06.7854322Z  * [new branch]                gh/kwen2501/211/head    -> origin/gh/kwen2501/211/head
2025-12-04T12:26:06.7854517Z  * [new branch]                gh/kwen2501/224/base    -> origin/gh/kwen2501/224/base
2025-12-04T12:26:06.7854588Z  * [new branch]                gh/kwen2501/224/head    -> origin/gh/kwen2501/224/head
2025-12-04T12:26:06.7854658Z  * [new branch]                gh/kwen2501/224/orig    -> origin/gh/kwen2501/224/orig
2025-12-04T12:26:06.7854730Z  * [new branch]                gh/kwen2501/228/base    -> origin/gh/kwen2501/228/base
2025-12-04T12:26:06.7854799Z  * [new branch]                gh/kwen2501/228/head    -> origin/gh/kwen2501/228/head
2025-12-04T12:26:06.7854867Z  * [new branch]                gh/kwen2501/228/orig    -> origin/gh/kwen2501/228/orig
2025-12-04T12:26:06.7854939Z  * [new branch]                gh/kwen2501/234/base    -> origin/gh/kwen2501/234/base
2025-12-04T12:26:06.7855009Z  * [new branch]                gh/kwen2501/234/head    -> origin/gh/kwen2501/234/head
2025-12-04T12:26:06.7855078Z  * [new branch]                gh/kwen2501/234/orig    -> origin/gh/kwen2501/234/orig
2025-12-04T12:26:06.7855150Z  * [new branch]                gh/kwen2501/235/base    -> origin/gh/kwen2501/235/base
2025-12-04T12:26:06.7855220Z  * [new branch]                gh/kwen2501/235/head    -> origin/gh/kwen2501/235/head
2025-12-04T12:26:06.7855289Z  * [new branch]                gh/kwen2501/235/orig    -> origin/gh/kwen2501/235/orig
2025-12-04T12:26:06.7855360Z  * [new branch]                gh/kwen2501/236/base    -> origin/gh/kwen2501/236/base
2025-12-04T12:26:06.7855429Z  * [new branch]                gh/kwen2501/236/head    -> origin/gh/kwen2501/236/head
2025-12-04T12:26:06.7855498Z  * [new branch]                gh/kwen2501/236/orig    -> origin/gh/kwen2501/236/orig
2025-12-04T12:26:06.7855571Z  * [new branch]                gh/kwen2501/237/base    -> origin/gh/kwen2501/237/base
2025-12-04T12:26:06.7855639Z  * [new branch]                gh/kwen2501/237/head    -> origin/gh/kwen2501/237/head
2025-12-04T12:26:06.7855707Z  * [new branch]                gh/kwen2501/237/orig    -> origin/gh/kwen2501/237/orig
2025-12-04T12:26:06.7855780Z  * [new branch]                gh/kwen2501/238/base    -> origin/gh/kwen2501/238/base
2025-12-04T12:26:06.7855849Z  * [new branch]                gh/kwen2501/238/head    -> origin/gh/kwen2501/238/head
2025-12-04T12:26:06.7855919Z  * [new branch]                gh/kwen2501/238/orig    -> origin/gh/kwen2501/238/orig
2025-12-04T12:26:06.7855988Z  * [new branch]                gh/kwen2501/240/base    -> origin/gh/kwen2501/240/base
2025-12-04T12:26:06.7856057Z  * [new branch]                gh/kwen2501/240/head    -> origin/gh/kwen2501/240/head
2025-12-04T12:26:06.7856126Z  * [new branch]                gh/kwen2501/240/orig    -> origin/gh/kwen2501/240/orig
2025-12-04T12:26:06.7856196Z  * [new branch]                gh/kwen2501/241/base    -> origin/gh/kwen2501/241/base
2025-12-04T12:26:06.7856267Z  * [new branch]                gh/kwen2501/241/head    -> origin/gh/kwen2501/241/head
2025-12-04T12:26:06.7856336Z  * [new branch]                gh/kwen2501/241/orig    -> origin/gh/kwen2501/241/orig
2025-12-04T12:26:06.7856405Z  * [new branch]                gh/kwen2501/247/base    -> origin/gh/kwen2501/247/base
2025-12-04T12:26:06.7856475Z  * [new branch]                gh/kwen2501/247/head    -> origin/gh/kwen2501/247/head
2025-12-04T12:26:06.7856546Z  * [new branch]                gh/kwen2501/247/orig    -> origin/gh/kwen2501/247/orig
2025-12-04T12:26:06.7856614Z  * [new branch]                gh/kwen2501/252/base    -> origin/gh/kwen2501/252/base
2025-12-04T12:26:06.7856683Z  * [new branch]                gh/kwen2501/252/head    -> origin/gh/kwen2501/252/head
2025-12-04T12:26:06.7856791Z  * [new branch]                gh/kwen2501/252/orig    -> origin/gh/kwen2501/252/orig
2025-12-04T12:26:06.7856905Z  * [new branch]                gh/kwen2501/259/base    -> origin/gh/kwen2501/259/base
2025-12-04T12:26:06.7856976Z  * [new branch]                gh/kwen2501/259/head    -> origin/gh/kwen2501/259/head
2025-12-04T12:26:06.7857049Z  * [new branch]                gh/kwen2501/259/orig    -> origin/gh/kwen2501/259/orig
2025-12-04T12:26:06.7857161Z  * [new branch]                gh/kwen2501/260/base    -> origin/gh/kwen2501/260/base
2025-12-04T12:26:06.7857229Z  * [new branch]                gh/kwen2501/260/head    -> origin/gh/kwen2501/260/head
2025-12-04T12:26:06.7857301Z  * [new branch]                gh/kwen2501/260/orig    -> origin/gh/kwen2501/260/orig
2025-12-04T12:26:06.7857370Z  * [new branch]                gh/kwen2501/268/base    -> origin/gh/kwen2501/268/base
2025-12-04T12:26:06.7857439Z  * [new branch]                gh/kwen2501/268/head    -> origin/gh/kwen2501/268/head
2025-12-04T12:26:06.7857513Z  * [new branch]                gh/kwen2501/268/orig    -> origin/gh/kwen2501/268/orig
2025-12-04T12:26:06.7857583Z  * [new branch]                gh/kwen2501/269/base    -> origin/gh/kwen2501/269/base
2025-12-04T12:26:06.7857655Z  * [new branch]                gh/kwen2501/269/head    -> origin/gh/kwen2501/269/head
2025-12-04T12:26:06.7857726Z  * [new branch]                gh/kwen2501/269/orig    -> origin/gh/kwen2501/269/orig
2025-12-04T12:26:06.7857797Z  * [new branch]                gh/kwen2501/270/base    -> origin/gh/kwen2501/270/base
2025-12-04T12:26:06.7857871Z  * [new branch]                gh/kwen2501/270/head    -> origin/gh/kwen2501/270/head
2025-12-04T12:26:06.7857940Z  * [new branch]                gh/kwen2501/270/orig    -> origin/gh/kwen2501/270/orig
2025-12-04T12:26:06.7858011Z  * [new branch]                gh/kwen2501/271/base    -> origin/gh/kwen2501/271/base
2025-12-04T12:26:06.7858082Z  * [new branch]                gh/kwen2501/271/head    -> origin/gh/kwen2501/271/head
2025-12-04T12:26:06.7858153Z  * [new branch]                gh/kwen2501/271/orig    -> origin/gh/kwen2501/271/orig
2025-12-04T12:26:06.7858224Z  * [new branch]                gh/kwen2501/274/base    -> origin/gh/kwen2501/274/base
2025-12-04T12:26:06.7858295Z  * [new branch]                gh/kwen2501/274/head    -> origin/gh/kwen2501/274/head
2025-12-04T12:26:06.7858366Z  * [new branch]                gh/kwen2501/274/orig    -> origin/gh/kwen2501/274/orig
2025-12-04T12:26:06.7858436Z  * [new branch]                gh/kwen2501/275/base    -> origin/gh/kwen2501/275/base
2025-12-04T12:26:06.7858508Z  * [new branch]                gh/kwen2501/275/head    -> origin/gh/kwen2501/275/head
2025-12-04T12:26:06.7858577Z  * [new branch]                gh/kwen2501/275/orig    -> origin/gh/kwen2501/275/orig
2025-12-04T12:26:06.7858645Z  * [new branch]                gh/kwen2501/276/base    -> origin/gh/kwen2501/276/base
2025-12-04T12:26:06.7858717Z  * [new branch]                gh/kwen2501/276/head    -> origin/gh/kwen2501/276/head
2025-12-04T12:26:06.7858786Z  * [new branch]                gh/kwen2501/276/orig    -> origin/gh/kwen2501/276/orig
2025-12-04T12:26:06.7858857Z  * [new branch]                gh/kwen2501/277/base    -> origin/gh/kwen2501/277/base
2025-12-04T12:26:06.7858931Z  * [new branch]                gh/kwen2501/277/head    -> origin/gh/kwen2501/277/head
2025-12-04T12:26:06.7859001Z  * [new branch]                gh/kwen2501/277/orig    -> origin/gh/kwen2501/277/orig
2025-12-04T12:26:06.7859072Z  * [new branch]                gh/kwen2501/278/base    -> origin/gh/kwen2501/278/base
2025-12-04T12:26:06.7859142Z  * [new branch]                gh/kwen2501/278/head    -> origin/gh/kwen2501/278/head
2025-12-04T12:26:06.7859212Z  * [new branch]                gh/kwen2501/278/orig    -> origin/gh/kwen2501/278/orig
2025-12-04T12:26:06.7859285Z  * [new branch]                gh/kwen2501/279/base    -> origin/gh/kwen2501/279/base
2025-12-04T12:26:06.7859356Z  * [new branch]                gh/kwen2501/279/head    -> origin/gh/kwen2501/279/head
2025-12-04T12:26:06.7859426Z  * [new branch]                gh/kwen2501/279/orig    -> origin/gh/kwen2501/279/orig
2025-12-04T12:26:06.7859522Z  * [new branch]                gh/kwen2501/280/base    -> origin/gh/kwen2501/280/base
2025-12-04T12:26:06.7859593Z  * [new branch]                gh/kwen2501/280/head    -> origin/gh/kwen2501/280/head
2025-12-04T12:26:06.7859685Z  * [new branch]                gh/kwen2501/280/orig    -> origin/gh/kwen2501/280/orig
2025-12-04T12:26:06.7859756Z  * [new branch]                gh/kwen2501/281/base    -> origin/gh/kwen2501/281/base
2025-12-04T12:26:06.7859827Z  * [new branch]                gh/kwen2501/281/head    -> origin/gh/kwen2501/281/head
2025-12-04T12:26:06.7859898Z  * [new branch]                gh/kwen2501/281/orig    -> origin/gh/kwen2501/281/orig
2025-12-04T12:26:06.7859970Z  * [new branch]                gh/kwen2501/282/base    -> origin/gh/kwen2501/282/base
2025-12-04T12:26:06.7860037Z  * [new branch]                gh/kwen2501/282/head    -> origin/gh/kwen2501/282/head
2025-12-04T12:26:06.7860107Z  * [new branch]                gh/kwen2501/282/orig    -> origin/gh/kwen2501/282/orig
2025-12-04T12:26:06.7860181Z  * [new branch]                gh/kwen2501/283/base    -> origin/gh/kwen2501/283/base
2025-12-04T12:26:06.7860250Z  * [new branch]                gh/kwen2501/283/head    -> origin/gh/kwen2501/283/head
2025-12-04T12:26:06.7860321Z  * [new branch]                gh/kwen2501/283/orig    -> origin/gh/kwen2501/283/orig
2025-12-04T12:26:06.7860392Z  * [new branch]                gh/kwen2501/284/base    -> origin/gh/kwen2501/284/base
2025-12-04T12:26:06.7860462Z  * [new branch]                gh/kwen2501/284/head    -> origin/gh/kwen2501/284/head
2025-12-04T12:26:06.7860534Z  * [new branch]                gh/kwen2501/284/orig    -> origin/gh/kwen2501/284/orig
2025-12-04T12:26:06.7860603Z  * [new branch]                gh/kwen2501/285/base    -> origin/gh/kwen2501/285/base
2025-12-04T12:26:06.7860673Z  * [new branch]                gh/kwen2501/285/head    -> origin/gh/kwen2501/285/head
2025-12-04T12:26:06.7860747Z  * [new branch]                gh/kwen2501/285/orig    -> origin/gh/kwen2501/285/orig
2025-12-04T12:26:06.7860817Z  * [new branch]                gh/kwen2501/286/base    -> origin/gh/kwen2501/286/base
2025-12-04T12:26:06.7860886Z  * [new branch]                gh/kwen2501/286/head    -> origin/gh/kwen2501/286/head
2025-12-04T12:26:06.7860959Z  * [new branch]                gh/kwen2501/286/orig    -> origin/gh/kwen2501/286/orig
2025-12-04T12:26:06.7861028Z  * [new branch]                gh/kwen2501/287/base    -> origin/gh/kwen2501/287/base
2025-12-04T12:26:06.7861097Z  * [new branch]                gh/kwen2501/287/head    -> origin/gh/kwen2501/287/head
2025-12-04T12:26:06.7861167Z  * [new branch]                gh/kwen2501/287/orig    -> origin/gh/kwen2501/287/orig
2025-12-04T12:26:06.7861236Z  * [new branch]                gh/kwen2501/288/base    -> origin/gh/kwen2501/288/base
2025-12-04T12:26:06.7861305Z  * [new branch]                gh/kwen2501/288/head    -> origin/gh/kwen2501/288/head
2025-12-04T12:26:06.7861376Z  * [new branch]                gh/kwen2501/288/orig    -> origin/gh/kwen2501/288/orig
2025-12-04T12:26:06.7861452Z  * [new branch]                gh/laithsakka/251/base  -> origin/gh/laithsakka/251/base
2025-12-04T12:26:06.7861528Z  * [new branch]                gh/laithsakka/251/head  -> origin/gh/laithsakka/251/head
2025-12-04T12:26:06.7861608Z  * [new branch]                gh/laithsakka/251/orig  -> origin/gh/laithsakka/251/orig
2025-12-04T12:26:06.7861682Z  * [new branch]                gh/laithsakka/276/base  -> origin/gh/laithsakka/276/base
2025-12-04T12:26:06.7861757Z  * [new branch]                gh/laithsakka/276/head  -> origin/gh/laithsakka/276/head
2025-12-04T12:26:06.7861833Z  * [new branch]                gh/laithsakka/276/orig  -> origin/gh/laithsakka/276/orig
2025-12-04T12:26:06.7861907Z  * [new branch]                gh/laithsakka/28/base   -> origin/gh/laithsakka/28/base
2025-12-04T12:26:06.7861981Z  * [new branch]                gh/laithsakka/29/base   -> origin/gh/laithsakka/29/base
2025-12-04T12:26:06.7862080Z  * [new branch]                gh/laithsakka/30/base   -> origin/gh/laithsakka/30/base
2025-12-04T12:26:06.7862155Z  * [new branch]                gh/laithsakka/30/head   -> origin/gh/laithsakka/30/head
2025-12-04T12:26:06.7862232Z  * [new branch]                gh/laithsakka/31/base   -> origin/gh/laithsakka/31/base
2025-12-04T12:26:06.7862329Z  * [new branch]                gh/laithsakka/31/head   -> origin/gh/laithsakka/31/head
2025-12-04T12:26:06.7862404Z  * [new branch]                gh/laithsakka/313/base  -> origin/gh/laithsakka/313/base
2025-12-04T12:26:06.7862482Z  * [new branch]                gh/laithsakka/313/head  -> origin/gh/laithsakka/313/head
2025-12-04T12:26:06.7862555Z  * [new branch]                gh/laithsakka/313/orig  -> origin/gh/laithsakka/313/orig
2025-12-04T12:26:06.7862627Z  * [new branch]                gh/laithsakka/316/base  -> origin/gh/laithsakka/316/base
2025-12-04T12:26:06.7862702Z  * [new branch]                gh/laithsakka/316/head  -> origin/gh/laithsakka/316/head
2025-12-04T12:26:06.7862777Z  * [new branch]                gh/laithsakka/316/orig  -> origin/gh/laithsakka/316/orig
2025-12-04T12:26:06.7862850Z  * [new branch]                gh/laithsakka/317/base  -> origin/gh/laithsakka/317/base
2025-12-04T12:26:06.7862929Z  * [new branch]                gh/laithsakka/317/head  -> origin/gh/laithsakka/317/head
2025-12-04T12:26:06.7863002Z  * [new branch]                gh/laithsakka/317/orig  -> origin/gh/laithsakka/317/orig
2025-12-04T12:26:06.7863075Z  * [new branch]                gh/laithsakka/319/base  -> origin/gh/laithsakka/319/base
2025-12-04T12:26:06.7863151Z  * [new branch]                gh/laithsakka/319/head  -> origin/gh/laithsakka/319/head
2025-12-04T12:26:06.7863225Z  * [new branch]                gh/laithsakka/319/orig  -> origin/gh/laithsakka/319/orig
2025-12-04T12:26:06.7863299Z  * [new branch]                gh/laithsakka/32/base   -> origin/gh/laithsakka/32/base
2025-12-04T12:26:06.7863377Z  * [new branch]                gh/laithsakka/32/head   -> origin/gh/laithsakka/32/head
2025-12-04T12:26:06.7863452Z  * [new branch]                gh/laithsakka/320/base  -> origin/gh/laithsakka/320/base
2025-12-04T12:26:06.7863525Z  * [new branch]                gh/laithsakka/320/head  -> origin/gh/laithsakka/320/head
2025-12-04T12:26:06.7863602Z  * [new branch]                gh/laithsakka/320/orig  -> origin/gh/laithsakka/320/orig
2025-12-04T12:26:06.7863674Z  * [new branch]                gh/laithsakka/321/base  -> origin/gh/laithsakka/321/base
2025-12-04T12:26:06.7863749Z  * [new branch]                gh/laithsakka/321/head  -> origin/gh/laithsakka/321/head
2025-12-04T12:26:06.7863823Z  * [new branch]                gh/laithsakka/321/orig  -> origin/gh/laithsakka/321/orig
2025-12-04T12:26:06.7863896Z  * [new branch]                gh/laithsakka/322/base  -> origin/gh/laithsakka/322/base
2025-12-04T12:26:06.7863972Z  * [new branch]                gh/laithsakka/322/head  -> origin/gh/laithsakka/322/head
2025-12-04T12:26:06.7864047Z  * [new branch]                gh/laithsakka/322/orig  -> origin/gh/laithsakka/322/orig
2025-12-04T12:26:06.7864121Z  * [new branch]                gh/laithsakka/323/base  -> origin/gh/laithsakka/323/base
2025-12-04T12:26:06.7864196Z  * [new branch]                gh/laithsakka/323/head  -> origin/gh/laithsakka/323/head
2025-12-04T12:26:06.7864272Z  * [new branch]                gh/laithsakka/323/orig  -> origin/gh/laithsakka/323/orig
2025-12-04T12:26:06.7864345Z  * [new branch]                gh/laithsakka/324/base  -> origin/gh/laithsakka/324/base
2025-12-04T12:26:06.7864425Z  * [new branch]                gh/laithsakka/324/head  -> origin/gh/laithsakka/324/head
2025-12-04T12:26:06.7864498Z  * [new branch]                gh/laithsakka/324/orig  -> origin/gh/laithsakka/324/orig
2025-12-04T12:26:06.7864570Z  * [new branch]                gh/laithsakka/325/base  -> origin/gh/laithsakka/325/base
2025-12-04T12:26:06.7864645Z  * [new branch]                gh/laithsakka/325/head  -> origin/gh/laithsakka/325/head
2025-12-04T12:26:06.7864743Z  * [new branch]                gh/laithsakka/325/orig  -> origin/gh/laithsakka/325/orig
2025-12-04T12:26:06.7864820Z  * [new branch]                gh/laithsakka/326/base  -> origin/gh/laithsakka/326/base
2025-12-04T12:26:06.7864895Z  * [new branch]                gh/laithsakka/326/head  -> origin/gh/laithsakka/326/head
2025-12-04T12:26:06.7864993Z  * [new branch]                gh/laithsakka/326/orig  -> origin/gh/laithsakka/326/orig
2025-12-04T12:26:06.7865066Z  * [new branch]                gh/laithsakka/327/base  -> origin/gh/laithsakka/327/base
2025-12-04T12:26:06.7865143Z  * [new branch]                gh/laithsakka/327/head  -> origin/gh/laithsakka/327/head
2025-12-04T12:26:06.7865218Z  * [new branch]                gh/laithsakka/327/orig  -> origin/gh/laithsakka/327/orig
2025-12-04T12:26:06.7865294Z  * [new branch]                gh/laithsakka/328/base  -> origin/gh/laithsakka/328/base
2025-12-04T12:26:06.7865368Z  * [new branch]                gh/laithsakka/328/head  -> origin/gh/laithsakka/328/head
2025-12-04T12:26:06.7865442Z  * [new branch]                gh/laithsakka/328/orig  -> origin/gh/laithsakka/328/orig
2025-12-04T12:26:06.7865515Z  * [new branch]                gh/liangel/4/base       -> origin/gh/liangel/4/base
2025-12-04T12:26:06.7865587Z  * [new branch]                gh/liangel/4/head       -> origin/gh/liangel/4/head
2025-12-04T12:26:06.7865656Z  * [new branch]                gh/liangel/4/orig       -> origin/gh/liangel/4/orig
2025-12-04T12:26:06.7865736Z  * [new branch]                gh/lucaskabela/1/base   -> origin/gh/lucaskabela/1/base
2025-12-04T12:26:06.7865812Z  * [new branch]                gh/lucaskabela/1/head   -> origin/gh/lucaskabela/1/head
2025-12-04T12:26:06.7865877Z  * [new branch]                gh/lw/4/base            -> origin/gh/lw/4/base
2025-12-04T12:26:06.7865943Z  * [new branch]                gh/lw/4/head            -> origin/gh/lw/4/head
2025-12-04T12:26:06.7866005Z  * [new branch]                gh/lw/4/orig            -> origin/gh/lw/4/orig
2025-12-04T12:26:06.7866068Z  * [new branch]                gh/lw/5/base            -> origin/gh/lw/5/base
2025-12-04T12:26:06.7866132Z  * [new branch]                gh/lw/5/head            -> origin/gh/lw/5/head
2025-12-04T12:26:06.7866194Z  * [new branch]                gh/lw/5/orig            -> origin/gh/lw/5/orig
2025-12-04T12:26:06.7866257Z  * [new branch]                gh/lw/6/base            -> origin/gh/lw/6/base
2025-12-04T12:26:06.7866320Z  * [new branch]                gh/lw/6/head            -> origin/gh/lw/6/head
2025-12-04T12:26:06.7866381Z  * [new branch]                gh/lw/6/orig            -> origin/gh/lw/6/orig
2025-12-04T12:26:06.7866450Z  * [new branch]                gh/malfet/14/base       -> origin/gh/malfet/14/base
2025-12-04T12:26:06.7866523Z  * [new branch]                gh/malfet/417/base      -> origin/gh/malfet/417/base
2025-12-04T12:26:06.7866593Z  * [new branch]                gh/malfet/417/head      -> origin/gh/malfet/417/head
2025-12-04T12:26:06.7866663Z  * [new branch]                gh/malfet/417/orig      -> origin/gh/malfet/417/orig
2025-12-04T12:26:06.7866736Z  * [new branch]                gh/malfet/506/base      -> origin/gh/malfet/506/base
2025-12-04T12:26:06.7866842Z  * [new branch]                gh/malfet/506/head      -> origin/gh/malfet/506/head
2025-12-04T12:26:06.7866913Z  * [new branch]                gh/malfet/506/orig      -> origin/gh/malfet/506/orig
2025-12-04T12:26:06.7866984Z  * [new branch]                gh/malfet/517/base      -> origin/gh/malfet/517/base
2025-12-04T12:26:06.7867053Z  * [new branch]                gh/malfet/517/head      -> origin/gh/malfet/517/head
2025-12-04T12:26:06.7867123Z  * [new branch]                gh/malfet/528/base      -> origin/gh/malfet/528/base
2025-12-04T12:26:06.7867191Z  * [new branch]                gh/malfet/528/head      -> origin/gh/malfet/528/head
2025-12-04T12:26:06.7867259Z  * [new branch]                gh/malfet/528/orig      -> origin/gh/malfet/528/orig
2025-12-04T12:26:06.7867329Z  * [new branch]                gh/malfet/537/base      -> origin/gh/malfet/537/base
2025-12-04T12:26:06.7867443Z  * [new branch]                gh/malfet/537/head      -> origin/gh/malfet/537/head
2025-12-04T12:26:06.7867512Z  * [new branch]                gh/malfet/537/orig      -> origin/gh/malfet/537/orig
2025-12-04T12:26:06.7867617Z  * [new branch]                gh/malfet/546/base      -> origin/gh/malfet/546/base
2025-12-04T12:26:06.7867685Z  * [new branch]                gh/malfet/546/head      -> origin/gh/malfet/546/head
2025-12-04T12:26:06.7867753Z  * [new branch]                gh/malfet/546/orig      -> origin/gh/malfet/546/orig
2025-12-04T12:26:06.7867823Z  * [new branch]                gh/malfet/565/base      -> origin/gh/malfet/565/base
2025-12-04T12:26:06.7867889Z  * [new branch]                gh/malfet/565/head      -> origin/gh/malfet/565/head
2025-12-04T12:26:06.7867956Z  * [new branch]                gh/malfet/565/orig      -> origin/gh/malfet/565/orig
2025-12-04T12:26:06.7868026Z  * [new branch]                gh/malfet/575/base      -> origin/gh/malfet/575/base
2025-12-04T12:26:06.7868095Z  * [new branch]                gh/malfet/575/head      -> origin/gh/malfet/575/head
2025-12-04T12:26:06.7868163Z  * [new branch]                gh/malfet/575/orig      -> origin/gh/malfet/575/orig
2025-12-04T12:26:06.7868235Z  * [new branch]                gh/malfet/580/base      -> origin/gh/malfet/580/base
2025-12-04T12:26:06.7868304Z  * [new branch]                gh/malfet/580/head      -> origin/gh/malfet/580/head
2025-12-04T12:26:06.7868371Z  * [new branch]                gh/malfet/580/orig      -> origin/gh/malfet/580/orig
2025-12-04T12:26:06.7868441Z  * [new branch]                gh/malfet/581/base      -> origin/gh/malfet/581/base
2025-12-04T12:26:06.7868509Z  * [new branch]                gh/malfet/581/head      -> origin/gh/malfet/581/head
2025-12-04T12:26:06.7868577Z  * [new branch]                gh/malfet/581/orig      -> origin/gh/malfet/581/orig
2025-12-04T12:26:06.7868646Z  * [new branch]                gh/malfet/583/base      -> origin/gh/malfet/583/base
2025-12-04T12:26:06.7868714Z  * [new branch]                gh/malfet/583/head      -> origin/gh/malfet/583/head
2025-12-04T12:26:06.7868783Z  * [new branch]                gh/malfet/583/orig      -> origin/gh/malfet/583/orig
2025-12-04T12:26:06.7868852Z  * [new branch]                gh/malfet/586/base      -> origin/gh/malfet/586/base
2025-12-04T12:26:06.7868921Z  * [new branch]                gh/malfet/586/head      -> origin/gh/malfet/586/head
2025-12-04T12:26:06.7868992Z  * [new branch]                gh/malfet/586/orig      -> origin/gh/malfet/586/orig
2025-12-04T12:26:06.7869060Z  * [new branch]                gh/malfet/587/base      -> origin/gh/malfet/587/base
2025-12-04T12:26:06.7869128Z  * [new branch]                gh/malfet/587/head      -> origin/gh/malfet/587/head
2025-12-04T12:26:06.7869197Z  * [new branch]                gh/malfet/587/orig      -> origin/gh/malfet/587/orig
2025-12-04T12:26:06.7869264Z  * [new branch]                gh/malfet/588/base      -> origin/gh/malfet/588/base
2025-12-04T12:26:06.7869333Z  * [new branch]                gh/malfet/588/head      -> origin/gh/malfet/588/head
2025-12-04T12:26:06.7869402Z  * [new branch]                gh/malfet/588/orig      -> origin/gh/malfet/588/orig
2025-12-04T12:26:06.7869471Z  * [new branch]                gh/malfet/589/base      -> origin/gh/malfet/589/base
2025-12-04T12:26:06.7869539Z  * [new branch]                gh/malfet/589/head      -> origin/gh/malfet/589/head
2025-12-04T12:26:06.7869607Z  * [new branch]                gh/malfet/589/orig      -> origin/gh/malfet/589/orig
2025-12-04T12:26:06.7869675Z  * [new branch]                gh/malfet/590/base      -> origin/gh/malfet/590/base
2025-12-04T12:26:06.7869743Z  * [new branch]                gh/malfet/590/head      -> origin/gh/malfet/590/head
2025-12-04T12:26:06.7869813Z  * [new branch]                gh/malfet/590/orig      -> origin/gh/malfet/590/orig
2025-12-04T12:26:06.7869882Z  * [new branch]                gh/malfet/591/base      -> origin/gh/malfet/591/base
2025-12-04T12:26:06.7869976Z  * [new branch]                gh/malfet/591/head      -> origin/gh/malfet/591/head
2025-12-04T12:26:06.7870048Z  * [new branch]                gh/malfet/591/orig      -> origin/gh/malfet/591/orig
2025-12-04T12:26:06.7870143Z  * [new branch]                gh/malfet/592/base      -> origin/gh/malfet/592/base
2025-12-04T12:26:06.7870210Z  * [new branch]                gh/malfet/592/head      -> origin/gh/malfet/592/head
2025-12-04T12:26:06.7870281Z  * [new branch]                gh/malfet/592/orig      -> origin/gh/malfet/592/orig
2025-12-04T12:26:06.7870349Z  * [new branch]                gh/malfet/593/base      -> origin/gh/malfet/593/base
2025-12-04T12:26:06.7870418Z  * [new branch]                gh/malfet/593/head      -> origin/gh/malfet/593/head
2025-12-04T12:26:06.7870486Z  * [new branch]                gh/malfet/593/orig      -> origin/gh/malfet/593/orig
2025-12-04T12:26:06.7870554Z  * [new branch]                gh/malfet/594/base      -> origin/gh/malfet/594/base
2025-12-04T12:26:06.7870625Z  * [new branch]                gh/malfet/594/head      -> origin/gh/malfet/594/head
2025-12-04T12:26:06.7870691Z  * [new branch]                gh/malfet/594/orig      -> origin/gh/malfet/594/orig
2025-12-04T12:26:06.7870761Z  * [new branch]                gh/malfet/595/base      -> origin/gh/malfet/595/base
2025-12-04T12:26:06.7870830Z  * [new branch]                gh/malfet/595/head      -> origin/gh/malfet/595/head
2025-12-04T12:26:06.7870898Z  * [new branch]                gh/malfet/595/orig      -> origin/gh/malfet/595/orig
2025-12-04T12:26:06.7870966Z  * [new branch]                gh/malfet/596/base      -> origin/gh/malfet/596/base
2025-12-04T12:26:06.7871037Z  * [new branch]                gh/malfet/596/head      -> origin/gh/malfet/596/head
2025-12-04T12:26:06.7871105Z  * [new branch]                gh/malfet/596/orig      -> origin/gh/malfet/596/orig
2025-12-04T12:26:06.7871173Z  * [new branch]                gh/malfet/597/base      -> origin/gh/malfet/597/base
2025-12-04T12:26:06.7871244Z  * [new branch]                gh/malfet/597/head      -> origin/gh/malfet/597/head
2025-12-04T12:26:06.7871312Z  * [new branch]                gh/malfet/597/orig      -> origin/gh/malfet/597/orig
2025-12-04T12:26:06.7871381Z  * [new branch]                gh/malfet/598/base      -> origin/gh/malfet/598/base
2025-12-04T12:26:06.7871452Z  * [new branch]                gh/malfet/598/head      -> origin/gh/malfet/598/head
2025-12-04T12:26:06.7871520Z  * [new branch]                gh/malfet/598/orig      -> origin/gh/malfet/598/orig
2025-12-04T12:26:06.7871588Z  * [new branch]                gh/malfet/599/base      -> origin/gh/malfet/599/base
2025-12-04T12:26:06.7871658Z  * [new branch]                gh/malfet/599/head      -> origin/gh/malfet/599/head
2025-12-04T12:26:06.7871726Z  * [new branch]                gh/malfet/599/orig      -> origin/gh/malfet/599/orig
2025-12-04T12:26:06.7871795Z  * [new branch]                gh/malfet/600/base      -> origin/gh/malfet/600/base
2025-12-04T12:26:06.7871866Z  * [new branch]                gh/malfet/600/head      -> origin/gh/malfet/600/head
2025-12-04T12:26:06.7871934Z  * [new branch]                gh/malfet/600/orig      -> origin/gh/malfet/600/orig
2025-12-04T12:26:06.7872002Z  * [new branch]                gh/malfet/601/base      -> origin/gh/malfet/601/base
2025-12-04T12:26:06.7872075Z  * [new branch]                gh/malfet/601/head      -> origin/gh/malfet/601/head
2025-12-04T12:26:06.7872142Z  * [new branch]                gh/malfet/601/orig      -> origin/gh/malfet/601/orig
2025-12-04T12:26:06.7872213Z  * [new branch]                gh/malfet/602/base      -> origin/gh/malfet/602/base
2025-12-04T12:26:06.7872282Z  * [new branch]                gh/malfet/602/head      -> origin/gh/malfet/602/head
2025-12-04T12:26:06.7872350Z  * [new branch]                gh/malfet/602/orig      -> origin/gh/malfet/602/orig
2025-12-04T12:26:06.7872419Z  * [new branch]                gh/malfet/603/base      -> origin/gh/malfet/603/base
2025-12-04T12:26:06.7872511Z  * [new branch]                gh/malfet/603/head      -> origin/gh/malfet/603/head
2025-12-04T12:26:06.7872580Z  * [new branch]                gh/malfet/603/orig      -> origin/gh/malfet/603/orig
2025-12-04T12:26:06.7872650Z  * [new branch]                gh/malfet/604/base      -> origin/gh/malfet/604/base
2025-12-04T12:26:06.7872742Z  * [new branch]                gh/malfet/604/head      -> origin/gh/malfet/604/head
2025-12-04T12:26:06.7872809Z  * [new branch]                gh/malfet/604/orig      -> origin/gh/malfet/604/orig
2025-12-04T12:26:06.7872878Z  * [new branch]                gh/malfet/605/base      -> origin/gh/malfet/605/base
2025-12-04T12:26:06.7872944Z  * [new branch]                gh/malfet/605/head      -> origin/gh/malfet/605/head
2025-12-04T12:26:06.7873010Z  * [new branch]                gh/malfet/605/orig      -> origin/gh/malfet/605/orig
2025-12-04T12:26:06.7873078Z  * [new branch]                gh/malfet/606/base      -> origin/gh/malfet/606/base
2025-12-04T12:26:06.7873145Z  * [new branch]                gh/malfet/606/head      -> origin/gh/malfet/606/head
2025-12-04T12:26:06.7873213Z  * [new branch]                gh/malfet/606/orig      -> origin/gh/malfet/606/orig
2025-12-04T12:26:06.7873281Z  * [new branch]                gh/malfet/607/base      -> origin/gh/malfet/607/base
2025-12-04T12:26:06.7873350Z  * [new branch]                gh/malfet/607/head      -> origin/gh/malfet/607/head
2025-12-04T12:26:06.7873417Z  * [new branch]                gh/malfet/607/orig      -> origin/gh/malfet/607/orig
2025-12-04T12:26:06.7873484Z  * [new branch]                gh/malfet/608/base      -> origin/gh/malfet/608/base
2025-12-04T12:26:06.7873551Z  * [new branch]                gh/malfet/608/head      -> origin/gh/malfet/608/head
2025-12-04T12:26:06.7873617Z  * [new branch]                gh/malfet/608/orig      -> origin/gh/malfet/608/orig
2025-12-04T12:26:06.7873685Z  * [new branch]                gh/malfet/609/base      -> origin/gh/malfet/609/base
2025-12-04T12:26:06.7873752Z  * [new branch]                gh/malfet/609/head      -> origin/gh/malfet/609/head
2025-12-04T12:26:06.7873821Z  * [new branch]                gh/malfet/609/orig      -> origin/gh/malfet/609/orig
2025-12-04T12:26:06.7873888Z  * [new branch]                gh/malfet/610/base      -> origin/gh/malfet/610/base
2025-12-04T12:26:06.7873958Z  * [new branch]                gh/malfet/610/head      -> origin/gh/malfet/610/head
2025-12-04T12:26:06.7874026Z  * [new branch]                gh/malfet/610/orig      -> origin/gh/malfet/610/orig
2025-12-04T12:26:06.7874093Z  * [new branch]                gh/malfet/611/base      -> origin/gh/malfet/611/base
2025-12-04T12:26:06.7874159Z  * [new branch]                gh/malfet/611/head      -> origin/gh/malfet/611/head
2025-12-04T12:26:06.7874227Z  * [new branch]                gh/malfet/611/orig      -> origin/gh/malfet/611/orig
2025-12-04T12:26:06.7874294Z  * [new branch]                gh/malfet/612/base      -> origin/gh/malfet/612/base
2025-12-04T12:26:06.7874362Z  * [new branch]                gh/malfet/612/head      -> origin/gh/malfet/612/head
2025-12-04T12:26:06.7874430Z  * [new branch]                gh/malfet/612/orig      -> origin/gh/malfet/612/orig
2025-12-04T12:26:06.7874497Z  * [new branch]                gh/malfet/64/base       -> origin/gh/malfet/64/base
2025-12-04T12:26:06.7874565Z  * [new branch]                gh/malfet/64/head       -> origin/gh/malfet/64/head
2025-12-04T12:26:06.7874656Z  * [new branch]                gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base
2025-12-04T12:26:06.7874743Z  * [new branch]                gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head
2025-12-04T12:26:06.7874826Z  * [new branch]                gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig
2025-12-04T12:26:06.7874897Z  * [new branch]                gh/markkm/1/base        -> origin/gh/markkm/1/base
2025-12-04T12:26:06.7874969Z  * [new branch]                gh/masnesral/1/base     -> origin/gh/masnesral/1/base
2025-12-04T12:26:06.7875066Z  * [new branch]                gh/masnesral/1/head     -> origin/gh/masnesral/1/head
2025-12-04T12:26:06.7875140Z  * [new branch]                gh/masnesral/1/orig     -> origin/gh/masnesral/1/orig
2025-12-04T12:26:06.7875211Z  * [new branch]                gh/mhorowitz/0/base     -> origin/gh/mhorowitz/0/base
2025-12-04T12:26:06.7875308Z  * [new branch]                gh/mhorowitz/0/head     -> origin/gh/mhorowitz/0/head
2025-12-04T12:26:06.7875380Z  * [new branch]                gh/mhorowitz/1/base     -> origin/gh/mhorowitz/1/base
2025-12-04T12:26:06.7875450Z  * [new branch]                gh/mhorowitz/1/head     -> origin/gh/mhorowitz/1/head
2025-12-04T12:26:06.7875521Z  * [new branch]                gh/mhorowitz/2/base     -> origin/gh/mhorowitz/2/base
2025-12-04T12:26:06.7875591Z  * [new branch]                gh/mhorowitz/2/head     -> origin/gh/mhorowitz/2/head
2025-12-04T12:26:06.7875661Z  * [new branch]                gh/mhorowitz/3/base     -> origin/gh/mhorowitz/3/base
2025-12-04T12:26:06.7875734Z  * [new branch]                gh/mhorowitz/3/head     -> origin/gh/mhorowitz/3/head
2025-12-04T12:26:06.7875804Z  * [new branch]                gh/mhorowitz/4/base     -> origin/gh/mhorowitz/4/base
2025-12-04T12:26:06.7875873Z  * [new branch]                gh/mhorowitz/4/head     -> origin/gh/mhorowitz/4/head
2025-12-04T12:26:06.7875946Z  * [new branch]                gh/mhorowitz/5/base     -> origin/gh/mhorowitz/5/base
2025-12-04T12:26:06.7876015Z  * [new branch]                gh/mhorowitz/5/head     -> origin/gh/mhorowitz/5/head
2025-12-04T12:26:06.7876084Z  * [new branch]                gh/mhorowitz/6/base     -> origin/gh/mhorowitz/6/base
2025-12-04T12:26:06.7876154Z  * [new branch]                gh/mhorowitz/6/head     -> origin/gh/mhorowitz/6/head
2025-12-04T12:26:06.7876255Z  * [new branch]                gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base
2025-12-04T12:26:06.7876351Z  * [new branch]                gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head
2025-12-04T12:26:06.7876446Z  * [new branch]                gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base
2025-12-04T12:26:06.7876538Z  * [new branch]                gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head
2025-12-04T12:26:06.7876631Z  * [new branch]                gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base
2025-12-04T12:26:06.7876723Z  * [new branch]                gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head
2025-12-04T12:26:06.7876866Z  * [new branch]                gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base
2025-12-04T12:26:06.7876959Z  * [new branch]                gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head
2025-12-04T12:26:06.7877051Z  * [new branch]                gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base
2025-12-04T12:26:06.7877142Z  * [new branch]                gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head
2025-12-04T12:26:06.7877234Z  * [new branch]                gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base
2025-12-04T12:26:06.7877325Z  * [new branch]                gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head
2025-12-04T12:26:06.7877417Z  * [new branch]                gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig
2025-12-04T12:26:06.7877508Z  * [new branch]                gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base
2025-12-04T12:26:06.7877599Z  * [new branch]                gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head
2025-12-04T12:26:06.7877690Z  * [new branch]                gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig
2025-12-04T12:26:06.7877782Z  * [new branch]                gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base
2025-12-04T12:26:06.7877873Z  * [new branch]                gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head
2025-12-04T12:26:06.7878010Z  * [new branch]                gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig
2025-12-04T12:26:06.7878104Z  * [new branch]                gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base
2025-12-04T12:26:06.7878230Z  * [new branch]                gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head
2025-12-04T12:26:06.7878321Z  * [new branch]                gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig
2025-12-04T12:26:06.7878413Z  * [new branch]                gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base
2025-12-04T12:26:06.7878505Z  * [new branch]                gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head
2025-12-04T12:26:06.7878599Z  * [new branch]                gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig
2025-12-04T12:26:06.7878691Z  * [new branch]                gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base
2025-12-04T12:26:06.7878784Z  * [new branch]                gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head
2025-12-04T12:26:06.7878879Z  * [new branch]                gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig
2025-12-04T12:26:06.7878973Z  * [new branch]                gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base
2025-12-04T12:26:06.7879065Z  * [new branch]                gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head
2025-12-04T12:26:06.7879158Z  * [new branch]                gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig
2025-12-04T12:26:06.7879249Z  * [new branch]                gh/mikaylagawarecki/351/base -> origin/gh/mikaylagawarecki/351/base
2025-12-04T12:26:06.7879340Z  * [new branch]                gh/mikaylagawarecki/351/head -> origin/gh/mikaylagawarecki/351/head
2025-12-04T12:26:06.7879436Z  * [new branch]                gh/mikaylagawarecki/351/orig -> origin/gh/mikaylagawarecki/351/orig
2025-12-04T12:26:06.7879528Z  * [new branch]                gh/mikaylagawarecki/352/base -> origin/gh/mikaylagawarecki/352/base
2025-12-04T12:26:06.7879621Z  * [new branch]                gh/mikaylagawarecki/352/head -> origin/gh/mikaylagawarecki/352/head
2025-12-04T12:26:06.7879717Z  * [new branch]                gh/mikaylagawarecki/352/orig -> origin/gh/mikaylagawarecki/352/orig
2025-12-04T12:26:06.7879809Z  * [new branch]                gh/mikaylagawarecki/353/base -> origin/gh/mikaylagawarecki/353/base
2025-12-04T12:26:06.7879903Z  * [new branch]                gh/mikaylagawarecki/353/head -> origin/gh/mikaylagawarecki/353/head
2025-12-04T12:26:06.7879995Z  * [new branch]                gh/mikaylagawarecki/353/orig -> origin/gh/mikaylagawarecki/353/orig
2025-12-04T12:26:06.7880086Z  * [new branch]                gh/mikaylagawarecki/354/base -> origin/gh/mikaylagawarecki/354/base
2025-12-04T12:26:06.7880180Z  * [new branch]                gh/mikaylagawarecki/354/head -> origin/gh/mikaylagawarecki/354/head
2025-12-04T12:26:06.7880275Z  * [new branch]                gh/mikaylagawarecki/354/orig -> origin/gh/mikaylagawarecki/354/orig
2025-12-04T12:26:06.7880366Z  * [new branch]                gh/mikaylagawarecki/356/base -> origin/gh/mikaylagawarecki/356/base
2025-12-04T12:26:06.7880461Z  * [new branch]                gh/mikaylagawarecki/356/head -> origin/gh/mikaylagawarecki/356/head
2025-12-04T12:26:06.7880552Z  * [new branch]                gh/mikaylagawarecki/356/orig -> origin/gh/mikaylagawarecki/356/orig
2025-12-04T12:26:06.7880643Z  * [new branch]                gh/mikaylagawarecki/357/base -> origin/gh/mikaylagawarecki/357/base
2025-12-04T12:26:06.7880737Z  * [new branch]                gh/mikaylagawarecki/357/head -> origin/gh/mikaylagawarecki/357/head
2025-12-04T12:26:06.7880829Z  * [new branch]                gh/mikaylagawarecki/357/orig -> origin/gh/mikaylagawarecki/357/orig
2025-12-04T12:26:06.7880921Z  * [new branch]                gh/mikaylagawarecki/359/base -> origin/gh/mikaylagawarecki/359/base
2025-12-04T12:26:06.7881042Z  * [new branch]                gh/mikaylagawarecki/359/head -> origin/gh/mikaylagawarecki/359/head
2025-12-04T12:26:06.7881136Z  * [new branch]                gh/mikaylagawarecki/359/orig -> origin/gh/mikaylagawarecki/359/orig
2025-12-04T12:26:06.7881248Z  * [new branch]                gh/mikaylagawarecki/360/base -> origin/gh/mikaylagawarecki/360/base
2025-12-04T12:26:06.7881343Z  * [new branch]                gh/mikaylagawarecki/360/head -> origin/gh/mikaylagawarecki/360/head
2025-12-04T12:26:06.7881434Z  * [new branch]                gh/mikaylagawarecki/360/orig -> origin/gh/mikaylagawarecki/360/orig
2025-12-04T12:26:06.7881527Z  * [new branch]                gh/mikaylagawarecki/361/base -> origin/gh/mikaylagawarecki/361/base
2025-12-04T12:26:06.7881619Z  * [new branch]                gh/mikaylagawarecki/361/head -> origin/gh/mikaylagawarecki/361/head
2025-12-04T12:26:06.7881711Z  * [new branch]                gh/mikaylagawarecki/361/orig -> origin/gh/mikaylagawarecki/361/orig
2025-12-04T12:26:06.7881809Z  * [new branch]                gh/mikaylagawarecki/362/base -> origin/gh/mikaylagawarecki/362/base
2025-12-04T12:26:06.7881904Z  * [new branch]                gh/mikaylagawarecki/362/head -> origin/gh/mikaylagawarecki/362/head
2025-12-04T12:26:06.7881999Z  * [new branch]                gh/mikaylagawarecki/362/orig -> origin/gh/mikaylagawarecki/362/orig
2025-12-04T12:26:06.7882090Z  * [new branch]                gh/mikaylagawarecki/363/base -> origin/gh/mikaylagawarecki/363/base
2025-12-04T12:26:06.7882180Z  * [new branch]                gh/mikaylagawarecki/363/head -> origin/gh/mikaylagawarecki/363/head
2025-12-04T12:26:06.7882270Z  * [new branch]                gh/mikaylagawarecki/363/orig -> origin/gh/mikaylagawarecki/363/orig
2025-12-04T12:26:06.7882365Z  * [new branch]                gh/mikaylagawarecki/364/base -> origin/gh/mikaylagawarecki/364/base
2025-12-04T12:26:06.7882457Z  * [new branch]                gh/mikaylagawarecki/364/head -> origin/gh/mikaylagawarecki/364/head
2025-12-04T12:26:06.7882549Z  * [new branch]                gh/mikaylagawarecki/364/orig -> origin/gh/mikaylagawarecki/364/orig
2025-12-04T12:26:06.7882643Z  * [new branch]                gh/mikaylagawarecki/365/base -> origin/gh/mikaylagawarecki/365/base
2025-12-04T12:26:06.7882737Z  * [new branch]                gh/mikaylagawarecki/365/head -> origin/gh/mikaylagawarecki/365/head
2025-12-04T12:26:06.7882830Z  * [new branch]                gh/mikaylagawarecki/365/orig -> origin/gh/mikaylagawarecki/365/orig
2025-12-04T12:26:06.7882922Z  * [new branch]                gh/mikaylagawarecki/366/base -> origin/gh/mikaylagawarecki/366/base
2025-12-04T12:26:06.7883014Z  * [new branch]                gh/mikaylagawarecki/366/head -> origin/gh/mikaylagawarecki/366/head
2025-12-04T12:26:06.7883110Z  * [new branch]                gh/mikaylagawarecki/366/orig -> origin/gh/mikaylagawarecki/366/orig
2025-12-04T12:26:06.7883203Z  * [new branch]                gh/mikaylagawarecki/367/base -> origin/gh/mikaylagawarecki/367/base
2025-12-04T12:26:06.7883295Z  * [new branch]                gh/mikaylagawarecki/367/head -> origin/gh/mikaylagawarecki/367/head
2025-12-04T12:26:06.7883389Z  * [new branch]                gh/mikaylagawarecki/367/orig -> origin/gh/mikaylagawarecki/367/orig
2025-12-04T12:26:06.7883482Z  * [new branch]                gh/mikaylagawarecki/368/base -> origin/gh/mikaylagawarecki/368/base
2025-12-04T12:26:06.7883574Z  * [new branch]                gh/mikaylagawarecki/368/head -> origin/gh/mikaylagawarecki/368/head
2025-12-04T12:26:06.7883669Z  * [new branch]                gh/mikaylagawarecki/368/orig -> origin/gh/mikaylagawarecki/368/orig
2025-12-04T12:26:06.7883760Z  * [new branch]                gh/mikaylagawarecki/369/base -> origin/gh/mikaylagawarecki/369/base
2025-12-04T12:26:06.7883851Z  * [new branch]                gh/mikaylagawarecki/369/head -> origin/gh/mikaylagawarecki/369/head
2025-12-04T12:26:06.7883946Z  * [new branch]                gh/mikaylagawarecki/369/orig -> origin/gh/mikaylagawarecki/369/orig
2025-12-04T12:26:06.7884063Z  * [new branch]                gh/mikaylagawarecki/370/base -> origin/gh/mikaylagawarecki/370/base
2025-12-04T12:26:06.7884157Z  * [new branch]                gh/mikaylagawarecki/370/head -> origin/gh/mikaylagawarecki/370/head
2025-12-04T12:26:06.7884272Z  * [new branch]                gh/mikaylagawarecki/370/orig -> origin/gh/mikaylagawarecki/370/orig
2025-12-04T12:26:06.7884364Z  * [new branch]                gh/mikaylagawarecki/371/base -> origin/gh/mikaylagawarecki/371/base
2025-12-04T12:26:06.7884458Z  * [new branch]                gh/mikaylagawarecki/371/head -> origin/gh/mikaylagawarecki/371/head
2025-12-04T12:26:06.7884549Z  * [new branch]                gh/mikaylagawarecki/371/orig -> origin/gh/mikaylagawarecki/371/orig
2025-12-04T12:26:06.7884640Z  * [new branch]                gh/mikaylagawarecki/372/base -> origin/gh/mikaylagawarecki/372/base
2025-12-04T12:26:06.7884734Z  * [new branch]                gh/mikaylagawarecki/372/head -> origin/gh/mikaylagawarecki/372/head
2025-12-04T12:26:06.7884827Z  * [new branch]                gh/mikaylagawarecki/372/orig -> origin/gh/mikaylagawarecki/372/orig
2025-12-04T12:26:06.7884918Z  * [new branch]                gh/mikaylagawarecki/373/base -> origin/gh/mikaylagawarecki/373/base
2025-12-04T12:26:06.7885016Z  * [new branch]                gh/mikaylagawarecki/373/head -> origin/gh/mikaylagawarecki/373/head
2025-12-04T12:26:06.7885110Z  * [new branch]                gh/mikaylagawarecki/373/orig -> origin/gh/mikaylagawarecki/373/orig
2025-12-04T12:26:06.7885202Z  * [new branch]                gh/mikaylagawarecki/374/base -> origin/gh/mikaylagawarecki/374/base
2025-12-04T12:26:06.7885297Z  * [new branch]                gh/mikaylagawarecki/374/head -> origin/gh/mikaylagawarecki/374/head
2025-12-04T12:26:06.7885390Z  * [new branch]                gh/mikaylagawarecki/374/orig -> origin/gh/mikaylagawarecki/374/orig
2025-12-04T12:26:06.7885481Z  * [new branch]                gh/mikaylagawarecki/375/base -> origin/gh/mikaylagawarecki/375/base
2025-12-04T12:26:06.7885576Z  * [new branch]                gh/mikaylagawarecki/375/head -> origin/gh/mikaylagawarecki/375/head
2025-12-04T12:26:06.7885667Z  * [new branch]                gh/mikaylagawarecki/375/orig -> origin/gh/mikaylagawarecki/375/orig
2025-12-04T12:26:06.7885763Z  * [new branch]                gh/mikaylagawarecki/376/base -> origin/gh/mikaylagawarecki/376/base
2025-12-04T12:26:06.7885856Z  * [new branch]                gh/mikaylagawarecki/376/head -> origin/gh/mikaylagawarecki/376/head
2025-12-04T12:26:06.7885947Z  * [new branch]                gh/mikaylagawarecki/376/orig -> origin/gh/mikaylagawarecki/376/orig
2025-12-04T12:26:06.7886041Z  * [new branch]                gh/mikaylagawarecki/377/base -> origin/gh/mikaylagawarecki/377/base
2025-12-04T12:26:06.7886132Z  * [new branch]                gh/mikaylagawarecki/377/head -> origin/gh/mikaylagawarecki/377/head
2025-12-04T12:26:06.7886224Z  * [new branch]                gh/mikaylagawarecki/377/orig -> origin/gh/mikaylagawarecki/377/orig
2025-12-04T12:26:06.7886319Z  * [new branch]                gh/mikaylagawarecki/378/base -> origin/gh/mikaylagawarecki/378/base
2025-12-04T12:26:06.7886411Z  * [new branch]                gh/mikaylagawarecki/378/head -> origin/gh/mikaylagawarecki/378/head
2025-12-04T12:26:06.7886504Z  * [new branch]                gh/mikaylagawarecki/378/orig -> origin/gh/mikaylagawarecki/378/orig
2025-12-04T12:26:06.7886600Z  * [new branch]                gh/mikaylagawarecki/379/base -> origin/gh/mikaylagawarecki/379/base
2025-12-04T12:26:06.7886692Z  * [new branch]                gh/mikaylagawarecki/379/head -> origin/gh/mikaylagawarecki/379/head
2025-12-04T12:26:06.7886848Z  * [new branch]                gh/mikaylagawarecki/379/orig -> origin/gh/mikaylagawarecki/379/orig
2025-12-04T12:26:06.7886944Z  * [new branch]                gh/mikaylagawarecki/380/base -> origin/gh/mikaylagawarecki/380/base
2025-12-04T12:26:06.7887036Z  * [new branch]                gh/mikaylagawarecki/380/head -> origin/gh/mikaylagawarecki/380/head
2025-12-04T12:26:06.7887162Z  * [new branch]                gh/mikaylagawarecki/380/orig -> origin/gh/mikaylagawarecki/380/orig
2025-12-04T12:26:06.7887257Z  * [new branch]                gh/mikaylagawarecki/381/base -> origin/gh/mikaylagawarecki/381/base
2025-12-04T12:26:06.7887392Z  * [new branch]                gh/mikaylagawarecki/381/head -> origin/gh/mikaylagawarecki/381/head
2025-12-04T12:26:06.7887486Z  * [new branch]                gh/mikaylagawarecki/381/orig -> origin/gh/mikaylagawarecki/381/orig
2025-12-04T12:26:06.7887578Z  * [new branch]                gh/mikaylagawarecki/382/base -> origin/gh/mikaylagawarecki/382/base
2025-12-04T12:26:06.7887669Z  * [new branch]                gh/mikaylagawarecki/382/head -> origin/gh/mikaylagawarecki/382/head
2025-12-04T12:26:06.7887763Z  * [new branch]                gh/mikaylagawarecki/382/orig -> origin/gh/mikaylagawarecki/382/orig
2025-12-04T12:26:06.7887855Z  * [new branch]                gh/mikaylagawarecki/383/base -> origin/gh/mikaylagawarecki/383/base
2025-12-04T12:26:06.7887948Z  * [new branch]                gh/mikaylagawarecki/383/head -> origin/gh/mikaylagawarecki/383/head
2025-12-04T12:26:06.7888044Z  * [new branch]                gh/mikaylagawarecki/383/orig -> origin/gh/mikaylagawarecki/383/orig
2025-12-04T12:26:06.7888137Z  * [new branch]                gh/mikaylagawarecki/384/base -> origin/gh/mikaylagawarecki/384/base
2025-12-04T12:26:06.7888229Z  * [new branch]                gh/mikaylagawarecki/384/head -> origin/gh/mikaylagawarecki/384/head
2025-12-04T12:26:06.7888324Z  * [new branch]                gh/mikaylagawarecki/384/orig -> origin/gh/mikaylagawarecki/384/orig
2025-12-04T12:26:06.7888416Z  * [new branch]                gh/mikaylagawarecki/385/base -> origin/gh/mikaylagawarecki/385/base
2025-12-04T12:26:06.7888510Z  * [new branch]                gh/mikaylagawarecki/385/head -> origin/gh/mikaylagawarecki/385/head
2025-12-04T12:26:06.7888605Z  * [new branch]                gh/mikaylagawarecki/385/orig -> origin/gh/mikaylagawarecki/385/orig
2025-12-04T12:26:06.7888698Z  * [new branch]                gh/mikaylagawarecki/386/base -> origin/gh/mikaylagawarecki/386/base
2025-12-04T12:26:06.7888791Z  * [new branch]                gh/mikaylagawarecki/386/head -> origin/gh/mikaylagawarecki/386/head
2025-12-04T12:26:06.7888883Z  * [new branch]                gh/mikaylagawarecki/386/orig -> origin/gh/mikaylagawarecki/386/orig
2025-12-04T12:26:06.7888975Z  * [new branch]                gh/mikaylagawarecki/387/base -> origin/gh/mikaylagawarecki/387/base
2025-12-04T12:26:06.7889069Z  * [new branch]                gh/mikaylagawarecki/387/head -> origin/gh/mikaylagawarecki/387/head
2025-12-04T12:26:06.7889161Z  * [new branch]                gh/mikaylagawarecki/387/orig -> origin/gh/mikaylagawarecki/387/orig
2025-12-04T12:26:06.7889253Z  * [new branch]                gh/mikaylagawarecki/388/base -> origin/gh/mikaylagawarecki/388/base
2025-12-04T12:26:06.7889350Z  * [new branch]                gh/mikaylagawarecki/388/head -> origin/gh/mikaylagawarecki/388/head
2025-12-04T12:26:06.7889444Z  * [new branch]                gh/mikaylagawarecki/388/orig -> origin/gh/mikaylagawarecki/388/orig
2025-12-04T12:26:06.7889537Z  * [new branch]                gh/mikaylagawarecki/389/base -> origin/gh/mikaylagawarecki/389/base
2025-12-04T12:26:06.7889633Z  * [new branch]                gh/mikaylagawarecki/389/head -> origin/gh/mikaylagawarecki/389/head
2025-12-04T12:26:06.7889724Z  * [new branch]                gh/mikaylagawarecki/389/orig -> origin/gh/mikaylagawarecki/389/orig
2025-12-04T12:26:06.7889816Z  * [new branch]                gh/mikaylagawarecki/390/base -> origin/gh/mikaylagawarecki/390/base
2025-12-04T12:26:06.7889909Z  * [new branch]                gh/mikaylagawarecki/390/head -> origin/gh/mikaylagawarecki/390/head
2025-12-04T12:26:06.7890001Z  * [new branch]                gh/mikaylagawarecki/390/orig -> origin/gh/mikaylagawarecki/390/orig
2025-12-04T12:26:06.7890095Z  * [new branch]                gh/mikaylagawarecki/391/base -> origin/gh/mikaylagawarecki/391/base
2025-12-04T12:26:06.7890209Z  * [new branch]                gh/mikaylagawarecki/391/head -> origin/gh/mikaylagawarecki/391/head
2025-12-04T12:26:06.7890304Z  * [new branch]                gh/mikaylagawarecki/391/orig -> origin/gh/mikaylagawarecki/391/orig
2025-12-04T12:26:06.7890420Z  * [new branch]                gh/mikaylagawarecki/392/base -> origin/gh/mikaylagawarecki/392/base
2025-12-04T12:26:06.7890513Z  * [new branch]                gh/mikaylagawarecki/392/head -> origin/gh/mikaylagawarecki/392/head
2025-12-04T12:26:06.7890603Z  * [new branch]                gh/mikaylagawarecki/392/orig -> origin/gh/mikaylagawarecki/392/orig
2025-12-04T12:26:06.7890698Z  * [new branch]                gh/mikaylagawarecki/393/base -> origin/gh/mikaylagawarecki/393/base
2025-12-04T12:26:06.7890789Z  * [new branch]                gh/mikaylagawarecki/393/head -> origin/gh/mikaylagawarecki/393/head
2025-12-04T12:26:06.7890881Z  * [new branch]                gh/mikaylagawarecki/393/orig -> origin/gh/mikaylagawarecki/393/orig
2025-12-04T12:26:06.7890956Z  * [new branch]                gh/mlazos/41/base       -> origin/gh/mlazos/41/base
2025-12-04T12:26:06.7891026Z  * [new branch]                gh/mlazos/41/head       -> origin/gh/mlazos/41/head
2025-12-04T12:26:06.7891097Z  * [new branch]                gh/mlazos/41/orig       -> origin/gh/mlazos/41/orig
2025-12-04T12:26:06.7891168Z  * [new branch]                gh/mlazos/42/base       -> origin/gh/mlazos/42/base
2025-12-04T12:26:06.7891236Z  * [new branch]                gh/mlazos/42/head       -> origin/gh/mlazos/42/head
2025-12-04T12:26:06.7891302Z  * [new branch]                gh/mlazos/42/orig       -> origin/gh/mlazos/42/orig
2025-12-04T12:26:06.7891372Z  * [new branch]                gh/mlazos/43/base       -> origin/gh/mlazos/43/base
2025-12-04T12:26:06.7891439Z  * [new branch]                gh/mlazos/43/head       -> origin/gh/mlazos/43/head
2025-12-04T12:26:06.7891505Z  * [new branch]                gh/mlazos/43/orig       -> origin/gh/mlazos/43/orig
2025-12-04T12:26:06.7891576Z  * [new branch]                gh/mlazos/44/base       -> origin/gh/mlazos/44/base
2025-12-04T12:26:06.7891644Z  * [new branch]                gh/mlazos/44/head       -> origin/gh/mlazos/44/head
2025-12-04T12:26:06.7891712Z  * [new branch]                gh/mlazos/44/orig       -> origin/gh/mlazos/44/orig
2025-12-04T12:26:06.7891781Z  * [new branch]                gh/mlazos/47/base       -> origin/gh/mlazos/47/base
2025-12-04T12:26:06.7891847Z  * [new branch]                gh/mlazos/47/head       -> origin/gh/mlazos/47/head
2025-12-04T12:26:06.7891916Z  * [new branch]                gh/mlazos/47/orig       -> origin/gh/mlazos/47/orig
2025-12-04T12:26:06.7891982Z  * [new branch]                gh/mlazos/48/base       -> origin/gh/mlazos/48/base
2025-12-04T12:26:06.7892049Z  * [new branch]                gh/mlazos/48/head       -> origin/gh/mlazos/48/head
2025-12-04T12:26:06.7892118Z  * [new branch]                gh/mlazos/48/orig       -> origin/gh/mlazos/48/orig
2025-12-04T12:26:06.7892186Z  * [new branch]                gh/mlazos/49/base       -> origin/gh/mlazos/49/base
2025-12-04T12:26:06.7892254Z  * [new branch]                gh/mlazos/49/head       -> origin/gh/mlazos/49/head
2025-12-04T12:26:06.7892323Z  * [new branch]                gh/mlazos/49/orig       -> origin/gh/mlazos/49/orig
2025-12-04T12:26:06.7892391Z  * [new branch]                gh/mlazos/50/base       -> origin/gh/mlazos/50/base
2025-12-04T12:26:06.7892457Z  * [new branch]                gh/mlazos/50/head       -> origin/gh/mlazos/50/head
2025-12-04T12:26:06.7892526Z  * [new branch]                gh/mlazos/50/orig       -> origin/gh/mlazos/50/orig
2025-12-04T12:26:06.7892592Z  * [new branch]                gh/mlazos/51/base       -> origin/gh/mlazos/51/base
2025-12-04T12:26:06.7892659Z  * [new branch]                gh/mlazos/51/head       -> origin/gh/mlazos/51/head
2025-12-04T12:26:06.7892729Z  * [new branch]                gh/mlazos/51/orig       -> origin/gh/mlazos/51/orig
2025-12-04T12:26:06.7892821Z  * [new branch]                gh/mlazos/52/base       -> origin/gh/mlazos/52/base
2025-12-04T12:26:06.7892889Z  * [new branch]                gh/mlazos/52/head       -> origin/gh/mlazos/52/head
2025-12-04T12:26:06.7892958Z  * [new branch]                gh/mlazos/52/orig       -> origin/gh/mlazos/52/orig
2025-12-04T12:26:06.7893050Z  * [new branch]                gh/mlazos/53/base       -> origin/gh/mlazos/53/base
2025-12-04T12:26:06.7893118Z  * [new branch]                gh/mlazos/53/head       -> origin/gh/mlazos/53/head
2025-12-04T12:26:06.7893188Z  * [new branch]                gh/mlazos/53/orig       -> origin/gh/mlazos/53/orig
2025-12-04T12:26:06.7893254Z  * [new branch]                gh/mlazos/54/base       -> origin/gh/mlazos/54/base
2025-12-04T12:26:06.7893323Z  * [new branch]                gh/mlazos/54/head       -> origin/gh/mlazos/54/head
2025-12-04T12:26:06.7893389Z  * [new branch]                gh/mlazos/54/orig       -> origin/gh/mlazos/54/orig
2025-12-04T12:26:06.7893457Z  * [new branch]                gh/mlazos/55/base       -> origin/gh/mlazos/55/base
2025-12-04T12:26:06.7893526Z  * [new branch]                gh/mlazos/55/head       -> origin/gh/mlazos/55/head
2025-12-04T12:26:06.7893595Z  * [new branch]                gh/mlazos/55/orig       -> origin/gh/mlazos/55/orig
2025-12-04T12:26:06.7893664Z  * [new branch]                gh/mlazos/56/base       -> origin/gh/mlazos/56/base
2025-12-04T12:26:06.7893733Z  * [new branch]                gh/mlazos/56/head       -> origin/gh/mlazos/56/head
2025-12-04T12:26:06.7893800Z  * [new branch]                gh/mlazos/56/orig       -> origin/gh/mlazos/56/orig
2025-12-04T12:26:06.7893866Z  * [new branch]                gh/mlazos/57/base       -> origin/gh/mlazos/57/base
2025-12-04T12:26:06.7893936Z  * [new branch]                gh/mlazos/57/head       -> origin/gh/mlazos/57/head
2025-12-04T12:26:06.7894002Z  * [new branch]                gh/mlazos/57/orig       -> origin/gh/mlazos/57/orig
2025-12-04T12:26:06.7894069Z  * [new branch]                gh/mlazos/58/base       -> origin/gh/mlazos/58/base
2025-12-04T12:26:06.7894139Z  * [new branch]                gh/mlazos/58/head       -> origin/gh/mlazos/58/head
2025-12-04T12:26:06.7894207Z  * [new branch]                gh/mlazos/58/orig       -> origin/gh/mlazos/58/orig
2025-12-04T12:26:06.7894275Z  * [new branch]                gh/mlazos/59/base       -> origin/gh/mlazos/59/base
2025-12-04T12:26:06.7894343Z  * [new branch]                gh/mlazos/59/head       -> origin/gh/mlazos/59/head
2025-12-04T12:26:06.7894411Z  * [new branch]                gh/mlazos/59/orig       -> origin/gh/mlazos/59/orig
2025-12-04T12:26:06.7894478Z  * [new branch]                gh/mlazos/60/base       -> origin/gh/mlazos/60/base
2025-12-04T12:26:06.7894545Z  * [new branch]                gh/mlazos/60/head       -> origin/gh/mlazos/60/head
2025-12-04T12:26:06.7894612Z  * [new branch]                gh/mlazos/60/orig       -> origin/gh/mlazos/60/orig
2025-12-04T12:26:06.7894677Z  * [new branch]                gh/mlazos/61/base       -> origin/gh/mlazos/61/base
2025-12-04T12:26:06.7894747Z  * [new branch]                gh/mlazos/61/head       -> origin/gh/mlazos/61/head
2025-12-04T12:26:06.7894814Z  * [new branch]                gh/mlazos/61/orig       -> origin/gh/mlazos/61/orig
2025-12-04T12:26:06.7894882Z  * [new branch]                gh/mlazos/62/base       -> origin/gh/mlazos/62/base
2025-12-04T12:26:06.7894951Z  * [new branch]                gh/mlazos/62/head       -> origin/gh/mlazos/62/head
2025-12-04T12:26:06.7895019Z  * [new branch]                gh/mlazos/62/orig       -> origin/gh/mlazos/62/orig
2025-12-04T12:26:06.7895088Z  * [new branch]                gh/mlazos/63/base       -> origin/gh/mlazos/63/base
2025-12-04T12:26:06.7895153Z  * [new branch]                gh/mlazos/63/head       -> origin/gh/mlazos/63/head
2025-12-04T12:26:06.7895220Z  * [new branch]                gh/mlazos/63/orig       -> origin/gh/mlazos/63/orig
2025-12-04T12:26:06.7895289Z  * [new branch]                gh/mlazos/64/base       -> origin/gh/mlazos/64/base
2025-12-04T12:26:06.7895381Z  * [new branch]                gh/mlazos/64/head       -> origin/gh/mlazos/64/head
2025-12-04T12:26:06.7895449Z  * [new branch]                gh/mlazos/64/orig       -> origin/gh/mlazos/64/orig
2025-12-04T12:26:06.7895542Z  * [new branch]                gh/mlazos/65/base       -> origin/gh/mlazos/65/base
2025-12-04T12:26:06.7895609Z  * [new branch]                gh/mlazos/65/head       -> origin/gh/mlazos/65/head
2025-12-04T12:26:06.7895674Z  * [new branch]                gh/mlazos/65/orig       -> origin/gh/mlazos/65/orig
2025-12-04T12:26:06.7895742Z  * [new branch]                gh/mlazos/66/base       -> origin/gh/mlazos/66/base
2025-12-04T12:26:06.7895809Z  * [new branch]                gh/mlazos/66/head       -> origin/gh/mlazos/66/head
2025-12-04T12:26:06.7895876Z  * [new branch]                gh/mlazos/66/orig       -> origin/gh/mlazos/66/orig
2025-12-04T12:26:06.7895946Z  * [new branch]                gh/mlazos/67/base       -> origin/gh/mlazos/67/base
2025-12-04T12:26:06.7896014Z  * [new branch]                gh/mlazos/67/head       -> origin/gh/mlazos/67/head
2025-12-04T12:26:06.7896080Z  * [new branch]                gh/mlazos/67/orig       -> origin/gh/mlazos/67/orig
2025-12-04T12:26:06.7896149Z  * [new branch]                gh/mlazos/68/base       -> origin/gh/mlazos/68/base
2025-12-04T12:26:06.7896219Z  * [new branch]                gh/mlazos/68/head       -> origin/gh/mlazos/68/head
2025-12-04T12:26:06.7896285Z  * [new branch]                gh/mlazos/68/orig       -> origin/gh/mlazos/68/orig
2025-12-04T12:26:06.7896358Z  * [new branch]                gh/mlazos/69/base       -> origin/gh/mlazos/69/base
2025-12-04T12:26:06.7896425Z  * [new branch]                gh/mlazos/69/head       -> origin/gh/mlazos/69/head
2025-12-04T12:26:06.7896491Z  * [new branch]                gh/mlazos/69/orig       -> origin/gh/mlazos/69/orig
2025-12-04T12:26:06.7896563Z  * [new branch]                gh/mlazos/70/base       -> origin/gh/mlazos/70/base
2025-12-04T12:26:06.7896635Z  * [new branch]                gh/mlazos/70/head       -> origin/gh/mlazos/70/head
2025-12-04T12:26:06.7896885Z  * [new branch]                gh/mlazos/70/orig       -> origin/gh/mlazos/70/orig
2025-12-04T12:26:06.7897068Z  * [new branch]                gh/mlazos/71/base       -> origin/gh/mlazos/71/base
2025-12-04T12:26:06.7897283Z  * [new branch]                gh/mlazos/71/head       -> origin/gh/mlazos/71/head
2025-12-04T12:26:06.7897463Z  * [new branch]                gh/mlazos/71/orig       -> origin/gh/mlazos/71/orig
2025-12-04T12:26:06.7897640Z  * [new branch]                gh/mlazos/72/base       -> origin/gh/mlazos/72/base
2025-12-04T12:26:06.7897818Z  * [new branch]                gh/mlazos/72/head       -> origin/gh/mlazos/72/head
2025-12-04T12:26:06.7897992Z  * [new branch]                gh/mlazos/72/orig       -> origin/gh/mlazos/72/orig
2025-12-04T12:26:06.7898162Z  * [new branch]                gh/mlazos/73/base       -> origin/gh/mlazos/73/base
2025-12-04T12:26:06.7898336Z  * [new branch]                gh/mlazos/73/head       -> origin/gh/mlazos/73/head
2025-12-04T12:26:06.7898509Z  * [new branch]                gh/mlazos/73/orig       -> origin/gh/mlazos/73/orig
2025-12-04T12:26:06.7898685Z  * [new branch]                gh/mrmiywj/1/base       -> origin/gh/mrmiywj/1/base
2025-12-04T12:26:06.7898990Z  * [new branch]                gh/mrmiywj/1/head       -> origin/gh/mrmiywj/1/head
2025-12-04T12:26:06.7899173Z  * [new branch]                gh/muchulee8/73/base    -> origin/gh/muchulee8/73/base
2025-12-04T12:26:06.7899360Z  * [new branch]                gh/muchulee8/73/head    -> origin/gh/muchulee8/73/head
2025-12-04T12:26:06.7899550Z  * [new branch]                gh/muchulee8/73/orig    -> origin/gh/muchulee8/73/orig
2025-12-04T12:26:06.7899754Z  * [new branch]                gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base
2025-12-04T12:26:06.7899964Z  * [new branch]                gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head
2025-12-04T12:26:06.7900208Z  * [new branch]                gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig
2025-12-04T12:26:06.7900409Z  * [new branch]                gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base
2025-12-04T12:26:06.7900618Z  * [new branch]                gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head
2025-12-04T12:26:06.7900874Z  * [new branch]                gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig
2025-12-04T12:26:06.7901079Z  * [new branch]                gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base
2025-12-04T12:26:06.7901276Z  * [new branch]                gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head
2025-12-04T12:26:06.7901474Z  * [new branch]                gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig
2025-12-04T12:26:06.7901674Z  * [new branch]                gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base
2025-12-04T12:26:06.7901870Z  * [new branch]                gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head
2025-12-04T12:26:06.7902073Z  * [new branch]                gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig
2025-12-04T12:26:06.7902271Z  * [new branch]                gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base
2025-12-04T12:26:06.7902475Z  * [new branch]                gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head
2025-12-04T12:26:06.7902675Z  * [new branch]                gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig
2025-12-04T12:26:06.7902873Z  * [new branch]                gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base
2025-12-04T12:26:06.7903073Z  * [new branch]                gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head
2025-12-04T12:26:06.7903273Z  * [new branch]                gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig
2025-12-04T12:26:06.7903474Z  * [new branch]                gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base
2025-12-04T12:26:06.7903681Z  * [new branch]                gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head
2025-12-04T12:26:06.7903894Z  * [new branch]                gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig
2025-12-04T12:26:06.7904104Z  * [new branch]                gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base
2025-12-04T12:26:06.7904306Z  * [new branch]                gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head
2025-12-04T12:26:06.7904507Z  * [new branch]                gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig
2025-12-04T12:26:06.7904715Z  * [new branch]                gh/naveenthangudu/9/base -> origin/gh/naveenthangudu/9/base
2025-12-04T12:26:06.7904926Z  * [new branch]                gh/naveenthangudu/9/head -> origin/gh/naveenthangudu/9/head
2025-12-04T12:26:06.7905127Z  * [new branch]                gh/naveenthangudu/9/orig -> origin/gh/naveenthangudu/9/orig
2025-12-04T12:26:06.7905323Z  * [new branch]                gh/nikitaved/1/base     -> origin/gh/nikitaved/1/base
2025-12-04T12:26:06.7905536Z  * [new branch]                gh/nikitaved/1/head     -> origin/gh/nikitaved/1/head
2025-12-04T12:26:06.7905730Z  * [new branch]                gh/nikitaved/1/orig     -> origin/gh/nikitaved/1/orig
2025-12-04T12:26:06.7905917Z  * [new branch]                gh/nikitaved/10/base    -> origin/gh/nikitaved/10/base
2025-12-04T12:26:06.7906110Z  * [new branch]                gh/nikitaved/10/head    -> origin/gh/nikitaved/10/head
2025-12-04T12:26:06.7906300Z  * [new branch]                gh/nikitaved/10/orig    -> origin/gh/nikitaved/10/orig
2025-12-04T12:26:06.7906483Z  * [new branch]                gh/nikitaved/11/base    -> origin/gh/nikitaved/11/base
2025-12-04T12:26:06.7906668Z  * [new branch]                gh/nikitaved/11/head    -> origin/gh/nikitaved/11/head
2025-12-04T12:26:06.7906960Z  * [new branch]                gh/nikitaved/11/orig    -> origin/gh/nikitaved/11/orig
2025-12-04T12:26:06.7907182Z  * [new branch]                gh/nikitaved/12/base    -> origin/gh/nikitaved/12/base
2025-12-04T12:26:06.7907457Z  * [new branch]                gh/nikitaved/12/head    -> origin/gh/nikitaved/12/head
2025-12-04T12:26:06.7907646Z  * [new branch]                gh/nikitaved/12/orig    -> origin/gh/nikitaved/12/orig
2025-12-04T12:26:06.7907876Z  * [new branch]                gh/nikitaved/13/base    -> origin/gh/nikitaved/13/base
2025-12-04T12:26:06.7908059Z  * [new branch]                gh/nikitaved/13/head    -> origin/gh/nikitaved/13/head
2025-12-04T12:26:06.7908244Z  * [new branch]                gh/nikitaved/13/orig    -> origin/gh/nikitaved/13/orig
2025-12-04T12:26:06.7908430Z  * [new branch]                gh/nikitaved/14/base    -> origin/gh/nikitaved/14/base
2025-12-04T12:26:06.7908613Z  * [new branch]                gh/nikitaved/14/head    -> origin/gh/nikitaved/14/head
2025-12-04T12:26:06.7908798Z  * [new branch]                gh/nikitaved/14/orig    -> origin/gh/nikitaved/14/orig
2025-12-04T12:26:06.7908986Z  * [new branch]                gh/nikitaved/15/base    -> origin/gh/nikitaved/15/base
2025-12-04T12:26:06.7909169Z  * [new branch]                gh/nikitaved/15/head    -> origin/gh/nikitaved/15/head
2025-12-04T12:26:06.7909354Z  * [new branch]                gh/nikitaved/15/orig    -> origin/gh/nikitaved/15/orig
2025-12-04T12:26:06.7909546Z  * [new branch]                gh/nikitaved/16/base    -> origin/gh/nikitaved/16/base
2025-12-04T12:26:06.7909728Z  * [new branch]                gh/nikitaved/16/head    -> origin/gh/nikitaved/16/head
2025-12-04T12:26:06.7909912Z  * [new branch]                gh/nikitaved/16/orig    -> origin/gh/nikitaved/16/orig
2025-12-04T12:26:06.7910097Z  * [new branch]                gh/nikitaved/2/base     -> origin/gh/nikitaved/2/base
2025-12-04T12:26:06.7910280Z  * [new branch]                gh/nikitaved/2/head     -> origin/gh/nikitaved/2/head
2025-12-04T12:26:06.7910463Z  * [new branch]                gh/nikitaved/2/orig     -> origin/gh/nikitaved/2/orig
2025-12-04T12:26:06.7910644Z  * [new branch]                gh/nikitaved/4/base     -> origin/gh/nikitaved/4/base
2025-12-04T12:26:06.7910833Z  * [new branch]                gh/nikitaved/4/head     -> origin/gh/nikitaved/4/head
2025-12-04T12:26:06.7911016Z  * [new branch]                gh/nikitaved/4/orig     -> origin/gh/nikitaved/4/orig
2025-12-04T12:26:06.7911200Z  * [new branch]                gh/nikitaved/5/base     -> origin/gh/nikitaved/5/base
2025-12-04T12:26:06.7911382Z  * [new branch]                gh/nikitaved/5/head     -> origin/gh/nikitaved/5/head
2025-12-04T12:26:06.7911564Z  * [new branch]                gh/nikitaved/5/orig     -> origin/gh/nikitaved/5/orig
2025-12-04T12:26:06.7911744Z  * [new branch]                gh/nikitaved/6/base     -> origin/gh/nikitaved/6/base
2025-12-04T12:26:06.7911926Z  * [new branch]                gh/nikitaved/6/head     -> origin/gh/nikitaved/6/head
2025-12-04T12:26:06.7912110Z  * [new branch]                gh/nikitaved/6/orig     -> origin/gh/nikitaved/6/orig
2025-12-04T12:26:06.7912288Z  * [new branch]                gh/nikitaved/8/base     -> origin/gh/nikitaved/8/base
2025-12-04T12:26:06.7912473Z  * [new branch]                gh/nikitaved/8/head     -> origin/gh/nikitaved/8/head
2025-12-04T12:26:06.7919611Z  * [new branch]                gh/nikitaved/8/orig     -> origin/gh/nikitaved/8/orig
2025-12-04T12:26:06.7919818Z  * [new branch]                gh/nikitaved/9/base     -> origin/gh/nikitaved/9/base
2025-12-04T12:26:06.7920025Z  * [new branch]                gh/nikitaved/9/head     -> origin/gh/nikitaved/9/head
2025-12-04T12:26:06.7920211Z  * [new branch]                gh/nikitaved/9/orig     -> origin/gh/nikitaved/9/orig
2025-12-04T12:26:06.7920392Z  * [new branch]                gh/oulgen/10/base       -> origin/gh/oulgen/10/base
2025-12-04T12:26:06.7920568Z  * [new branch]                gh/oulgen/10/head       -> origin/gh/oulgen/10/head
2025-12-04T12:26:06.7920742Z  * [new branch]                gh/oulgen/10/orig       -> origin/gh/oulgen/10/orig
2025-12-04T12:26:06.7920918Z  * [new branch]                gh/oulgen/11/base       -> origin/gh/oulgen/11/base
2025-12-04T12:26:06.7921155Z  * [new branch]                gh/oulgen/11/head       -> origin/gh/oulgen/11/head
2025-12-04T12:26:06.7921334Z  * [new branch]                gh/oulgen/11/orig       -> origin/gh/oulgen/11/orig
2025-12-04T12:26:06.7921546Z  * [new branch]                gh/oulgen/12/base       -> origin/gh/oulgen/12/base
2025-12-04T12:26:06.7921718Z  * [new branch]                gh/oulgen/12/head       -> origin/gh/oulgen/12/head
2025-12-04T12:26:06.7921892Z  * [new branch]                gh/oulgen/12/orig       -> origin/gh/oulgen/12/orig
2025-12-04T12:26:06.7922105Z  * [new branch]                gh/oulgen/13/base       -> origin/gh/oulgen/13/base
2025-12-04T12:26:06.7922278Z  * [new branch]                gh/oulgen/13/head       -> origin/gh/oulgen/13/head
2025-12-04T12:26:06.7922448Z  * [new branch]                gh/oulgen/13/orig       -> origin/gh/oulgen/13/orig
2025-12-04T12:26:06.7922619Z  * [new branch]                gh/oulgen/14/base       -> origin/gh/oulgen/14/base
2025-12-04T12:26:06.7922798Z  * [new branch]                gh/oulgen/14/head       -> origin/gh/oulgen/14/head
2025-12-04T12:26:06.7922970Z  * [new branch]                gh/oulgen/14/orig       -> origin/gh/oulgen/14/orig
2025-12-04T12:26:06.7923150Z  * [new branch]                gh/oulgen/15/base       -> origin/gh/oulgen/15/base
2025-12-04T12:26:06.7923326Z  * [new branch]                gh/oulgen/15/head       -> origin/gh/oulgen/15/head
2025-12-04T12:26:06.7923498Z  * [new branch]                gh/oulgen/15/orig       -> origin/gh/oulgen/15/orig
2025-12-04T12:26:06.7923669Z  * [new branch]                gh/oulgen/16/base       -> origin/gh/oulgen/16/base
2025-12-04T12:26:06.7923843Z  * [new branch]                gh/oulgen/16/head       -> origin/gh/oulgen/16/head
2025-12-04T12:26:06.7924017Z  * [new branch]                gh/oulgen/16/orig       -> origin/gh/oulgen/16/orig
2025-12-04T12:26:06.7924191Z  * [new branch]                gh/oulgen/17/base       -> origin/gh/oulgen/17/base
2025-12-04T12:26:06.7924368Z  * [new branch]                gh/oulgen/17/head       -> origin/gh/oulgen/17/head
2025-12-04T12:26:06.7924542Z  * [new branch]                gh/oulgen/17/orig       -> origin/gh/oulgen/17/orig
2025-12-04T12:26:06.7924720Z  * [new branch]                gh/oulgen/18/base       -> origin/gh/oulgen/18/base
2025-12-04T12:26:06.7924892Z  * [new branch]                gh/oulgen/18/head       -> origin/gh/oulgen/18/head
2025-12-04T12:26:06.7925068Z  * [new branch]                gh/oulgen/18/orig       -> origin/gh/oulgen/18/orig
2025-12-04T12:26:06.7925241Z  * [new branch]                gh/oulgen/19/base       -> origin/gh/oulgen/19/base
2025-12-04T12:26:06.7925412Z  * [new branch]                gh/oulgen/19/head       -> origin/gh/oulgen/19/head
2025-12-04T12:26:06.7925587Z  * [new branch]                gh/oulgen/19/orig       -> origin/gh/oulgen/19/orig
2025-12-04T12:26:06.7925762Z  * [new branch]                gh/oulgen/20/base       -> origin/gh/oulgen/20/base
2025-12-04T12:26:06.7925938Z  * [new branch]                gh/oulgen/20/head       -> origin/gh/oulgen/20/head
2025-12-04T12:26:06.7926112Z  * [new branch]                gh/oulgen/20/orig       -> origin/gh/oulgen/20/orig
2025-12-04T12:26:06.7926287Z  * [new branch]                gh/oulgen/21/base       -> origin/gh/oulgen/21/base
2025-12-04T12:26:06.7926463Z  * [new branch]                gh/oulgen/21/head       -> origin/gh/oulgen/21/head
2025-12-04T12:26:06.7926636Z  * [new branch]                gh/oulgen/21/orig       -> origin/gh/oulgen/21/orig
2025-12-04T12:26:06.7926852Z  * [new branch]                gh/oulgen/22/base       -> origin/gh/oulgen/22/base
2025-12-04T12:26:06.7927025Z  * [new branch]                gh/oulgen/22/head       -> origin/gh/oulgen/22/head
2025-12-04T12:26:06.7927197Z  * [new branch]                gh/oulgen/22/orig       -> origin/gh/oulgen/22/orig
2025-12-04T12:26:06.7927369Z  * [new branch]                gh/oulgen/23/base       -> origin/gh/oulgen/23/base
2025-12-04T12:26:06.7927590Z  * [new branch]                gh/oulgen/23/head       -> origin/gh/oulgen/23/head
2025-12-04T12:26:06.7927767Z  * [new branch]                gh/oulgen/23/orig       -> origin/gh/oulgen/23/orig
2025-12-04T12:26:06.7927939Z  * [new branch]                gh/oulgen/24/base       -> origin/gh/oulgen/24/base
2025-12-04T12:26:06.7928155Z  * [new branch]                gh/oulgen/24/head       -> origin/gh/oulgen/24/head
2025-12-04T12:26:06.7928330Z  * [new branch]                gh/oulgen/24/orig       -> origin/gh/oulgen/24/orig
2025-12-04T12:26:06.7928503Z  * [new branch]                gh/oulgen/25/base       -> origin/gh/oulgen/25/base
2025-12-04T12:26:06.7928677Z  * [new branch]                gh/oulgen/25/head       -> origin/gh/oulgen/25/head
2025-12-04T12:26:06.7928849Z  * [new branch]                gh/oulgen/25/orig       -> origin/gh/oulgen/25/orig
2025-12-04T12:26:06.7929021Z  * [new branch]                gh/oulgen/26/base       -> origin/gh/oulgen/26/base
2025-12-04T12:26:06.7929198Z  * [new branch]                gh/oulgen/26/head       -> origin/gh/oulgen/26/head
2025-12-04T12:26:06.7929370Z  * [new branch]                gh/oulgen/26/orig       -> origin/gh/oulgen/26/orig
2025-12-04T12:26:06.7929542Z  * [new branch]                gh/oulgen/4/base        -> origin/gh/oulgen/4/base
2025-12-04T12:26:06.7929718Z  * [new branch]                gh/oulgen/4/head        -> origin/gh/oulgen/4/head
2025-12-04T12:26:06.7929891Z  * [new branch]                gh/oulgen/4/orig        -> origin/gh/oulgen/4/orig
2025-12-04T12:26:06.7930062Z  * [new branch]                gh/oulgen/7/base        -> origin/gh/oulgen/7/base
2025-12-04T12:26:06.7930232Z  * [new branch]                gh/oulgen/7/head        -> origin/gh/oulgen/7/head
2025-12-04T12:26:06.7930400Z  * [new branch]                gh/oulgen/7/orig        -> origin/gh/oulgen/7/orig
2025-12-04T12:26:06.7930569Z  * [new branch]                gh/oulgen/8/base        -> origin/gh/oulgen/8/base
2025-12-04T12:26:06.7930738Z  * [new branch]                gh/oulgen/8/head        -> origin/gh/oulgen/8/head
2025-12-04T12:26:06.7930911Z  * [new branch]                gh/oulgen/8/orig        -> origin/gh/oulgen/8/orig
2025-12-04T12:26:06.7931082Z  * [new branch]                gh/oulgen/9/base        -> origin/gh/oulgen/9/base
2025-12-04T12:26:06.7931255Z  * [new branch]                gh/oulgen/9/head        -> origin/gh/oulgen/9/head
2025-12-04T12:26:06.7931441Z  * [new branch]                gh/oulgen/9/orig        -> origin/gh/oulgen/9/orig
2025-12-04T12:26:06.7931650Z  * [new branch]                gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization
2025-12-04T12:26:06.7931861Z  * [new branch]                gh/pearu/108/base       -> origin/gh/pearu/108/base
2025-12-04T12:26:06.7932035Z  * [new branch]                gh/pearu/108/head       -> origin/gh/pearu/108/head
2025-12-04T12:26:06.7932208Z  * [new branch]                gh/pearu/108/orig       -> origin/gh/pearu/108/orig
2025-12-04T12:26:06.7932379Z  * [new branch]                gh/pearu/109/base       -> origin/gh/pearu/109/base
2025-12-04T12:26:06.7932551Z  * [new branch]                gh/pearu/109/head       -> origin/gh/pearu/109/head
2025-12-04T12:26:06.7932723Z  * [new branch]                gh/pearu/109/orig       -> origin/gh/pearu/109/orig
2025-12-04T12:26:06.7932895Z  * [new branch]                gh/pearu/110/base       -> origin/gh/pearu/110/base
2025-12-04T12:26:06.7933064Z  * [new branch]                gh/pearu/110/head       -> origin/gh/pearu/110/head
2025-12-04T12:26:06.7933235Z  * [new branch]                gh/pearu/110/orig       -> origin/gh/pearu/110/orig
2025-12-04T12:26:06.7933409Z  * [new branch]                gh/pearu/111/base       -> origin/gh/pearu/111/base
2025-12-04T12:26:06.7933581Z  * [new branch]                gh/pearu/111/head       -> origin/gh/pearu/111/head
2025-12-04T12:26:06.7933752Z  * [new branch]                gh/pearu/111/orig       -> origin/gh/pearu/111/orig
2025-12-04T12:26:06.7933923Z  * [new branch]                gh/pearu/112/base       -> origin/gh/pearu/112/base
2025-12-04T12:26:06.7934141Z  * [new branch]                gh/pearu/112/head       -> origin/gh/pearu/112/head
2025-12-04T12:26:06.7934312Z  * [new branch]                gh/pearu/112/orig       -> origin/gh/pearu/112/orig
2025-12-04T12:26:06.7934516Z  * [new branch]                gh/pearu/115/base       -> origin/gh/pearu/115/base
2025-12-04T12:26:06.7934688Z  * [new branch]                gh/pearu/115/head       -> origin/gh/pearu/115/head
2025-12-04T12:26:06.7934859Z  * [new branch]                gh/pearu/115/orig       -> origin/gh/pearu/115/orig
2025-12-04T12:26:06.7935029Z  * [new branch]                gh/pearu/116/base       -> origin/gh/pearu/116/base
2025-12-04T12:26:06.7935200Z  * [new branch]                gh/pearu/116/head       -> origin/gh/pearu/116/head
2025-12-04T12:26:06.7935374Z  * [new branch]                gh/pearu/116/orig       -> origin/gh/pearu/116/orig
2025-12-04T12:26:06.7935544Z  * [new branch]                gh/pearu/117/base       -> origin/gh/pearu/117/base
2025-12-04T12:26:06.7935719Z  * [new branch]                gh/pearu/117/head       -> origin/gh/pearu/117/head
2025-12-04T12:26:06.7935890Z  * [new branch]                gh/pearu/117/orig       -> origin/gh/pearu/117/orig
2025-12-04T12:26:06.7936065Z  * [new branch]                gh/pearu/118/base       -> origin/gh/pearu/118/base
2025-12-04T12:26:06.7936240Z  * [new branch]                gh/pearu/118/head       -> origin/gh/pearu/118/head
2025-12-04T12:26:06.7936408Z  * [new branch]                gh/pearu/118/orig       -> origin/gh/pearu/118/orig
2025-12-04T12:26:06.7936580Z  * [new branch]                gh/pearu/119/base       -> origin/gh/pearu/119/base
2025-12-04T12:26:06.7936804Z  * [new branch]                gh/pearu/119/head       -> origin/gh/pearu/119/head
2025-12-04T12:26:06.7936979Z  * [new branch]                gh/pearu/119/orig       -> origin/gh/pearu/119/orig
2025-12-04T12:26:06.7937149Z  * [new branch]                gh/pearu/139/base       -> origin/gh/pearu/139/base
2025-12-04T12:26:06.7937324Z  * [new branch]                gh/pearu/139/head       -> origin/gh/pearu/139/head
2025-12-04T12:26:06.7937492Z  * [new branch]                gh/pearu/139/orig       -> origin/gh/pearu/139/orig
2025-12-04T12:26:06.7937664Z  * [new branch]                gh/pearu/140/base       -> origin/gh/pearu/140/base
2025-12-04T12:26:06.7937840Z  * [new branch]                gh/pearu/140/head       -> origin/gh/pearu/140/head
2025-12-04T12:26:06.7938010Z  * [new branch]                gh/pearu/140/orig       -> origin/gh/pearu/140/orig
2025-12-04T12:26:06.7938185Z  * [new branch]                gh/pearu/142/base       -> origin/gh/pearu/142/base
2025-12-04T12:26:06.7938358Z  * [new branch]                gh/pearu/142/head       -> origin/gh/pearu/142/head
2025-12-04T12:26:06.7938527Z  * [new branch]                gh/pearu/142/orig       -> origin/gh/pearu/142/orig
2025-12-04T12:26:06.7938700Z  * [new branch]                gh/pearu/143/base       -> origin/gh/pearu/143/base
2025-12-04T12:26:06.7938874Z  * [new branch]                gh/pearu/143/head       -> origin/gh/pearu/143/head
2025-12-04T12:26:06.7939044Z  * [new branch]                gh/pearu/143/orig       -> origin/gh/pearu/143/orig
2025-12-04T12:26:06.7939215Z  * [new branch]                gh/pearu/147/base       -> origin/gh/pearu/147/base
2025-12-04T12:26:06.7939390Z  * [new branch]                gh/pearu/147/head       -> origin/gh/pearu/147/head
2025-12-04T12:26:06.7939562Z  * [new branch]                gh/pearu/147/orig       -> origin/gh/pearu/147/orig
2025-12-04T12:26:06.7939735Z  * [new branch]                gh/pearu/149/base       -> origin/gh/pearu/149/base
2025-12-04T12:26:06.7939907Z  * [new branch]                gh/pearu/149/head       -> origin/gh/pearu/149/head
2025-12-04T12:26:06.7940078Z  * [new branch]                gh/pearu/149/orig       -> origin/gh/pearu/149/orig
2025-12-04T12:26:06.7940254Z  * [new branch]                gh/pearu/150/base       -> origin/gh/pearu/150/base
2025-12-04T12:26:06.7940469Z  * [new branch]                gh/pearu/150/head       -> origin/gh/pearu/150/head
2025-12-04T12:26:06.7940642Z  * [new branch]                gh/pearu/150/orig       -> origin/gh/pearu/150/orig
2025-12-04T12:26:06.7940813Z  * [new branch]                gh/pearu/151/base       -> origin/gh/pearu/151/base
2025-12-04T12:26:06.7941029Z  * [new branch]                gh/pearu/151/head       -> origin/gh/pearu/151/head
2025-12-04T12:26:06.7941200Z  * [new branch]                gh/pearu/151/orig       -> origin/gh/pearu/151/orig
2025-12-04T12:26:06.7941373Z  * [new branch]                gh/pearu/152/base       -> origin/gh/pearu/152/base
2025-12-04T12:26:06.7941543Z  * [new branch]                gh/pearu/152/head       -> origin/gh/pearu/152/head
2025-12-04T12:26:06.7941714Z  * [new branch]                gh/pearu/152/orig       -> origin/gh/pearu/152/orig
2025-12-04T12:26:06.7941884Z  * [new branch]                gh/pearu/153/base       -> origin/gh/pearu/153/base
2025-12-04T12:26:06.7942058Z  * [new branch]                gh/pearu/153/head       -> origin/gh/pearu/153/head
2025-12-04T12:26:06.7942230Z  * [new branch]                gh/pearu/153/orig       -> origin/gh/pearu/153/orig
2025-12-04T12:26:06.7942404Z  * [new branch]                gh/pearu/154/base       -> origin/gh/pearu/154/base
2025-12-04T12:26:06.7942578Z  * [new branch]                gh/pearu/154/head       -> origin/gh/pearu/154/head
2025-12-04T12:26:06.7942752Z  * [new branch]                gh/pearu/154/orig       -> origin/gh/pearu/154/orig
2025-12-04T12:26:06.7942924Z  * [new branch]                gh/pearu/155/base       -> origin/gh/pearu/155/base
2025-12-04T12:26:06.7943095Z  * [new branch]                gh/pearu/155/head       -> origin/gh/pearu/155/head
2025-12-04T12:26:06.7943266Z  * [new branch]                gh/pearu/155/orig       -> origin/gh/pearu/155/orig
2025-12-04T12:26:06.7943437Z  * [new branch]                gh/pearu/156/base       -> origin/gh/pearu/156/base
2025-12-04T12:26:06.7943609Z  * [new branch]                gh/pearu/156/head       -> origin/gh/pearu/156/head
2025-12-04T12:26:06.7943784Z  * [new branch]                gh/pearu/156/orig       -> origin/gh/pearu/156/orig
2025-12-04T12:26:06.7943959Z  * [new branch]                gh/pearu/56/base        -> origin/gh/pearu/56/base
2025-12-04T12:26:06.7944135Z  * [new branch]                gh/pearu/56/head        -> origin/gh/pearu/56/head
2025-12-04T12:26:06.7944305Z  * [new branch]                gh/pearu/56/orig        -> origin/gh/pearu/56/orig
2025-12-04T12:26:06.7944474Z  * [new branch]                gh/pearu/97/base        -> origin/gh/pearu/97/base
2025-12-04T12:26:06.7944645Z  * [new branch]                gh/pearu/97/head        -> origin/gh/pearu/97/head
2025-12-04T12:26:06.7944814Z  * [new branch]                gh/pearu/97/orig        -> origin/gh/pearu/97/orig
2025-12-04T12:26:06.7944994Z  * [new branch]                gh/pianpwk/21/base      -> origin/gh/pianpwk/21/base
2025-12-04T12:26:06.7945180Z  * [new branch]                gh/pianpwk/21/head      -> origin/gh/pianpwk/21/head
2025-12-04T12:26:06.7945363Z  * [new branch]                gh/pianpwk/28/base      -> origin/gh/pianpwk/28/base
2025-12-04T12:26:06.7945540Z  * [new branch]                gh/pianpwk/28/head      -> origin/gh/pianpwk/28/head
2025-12-04T12:26:06.7945723Z  * [new branch]                gh/pianpwk/28/orig      -> origin/gh/pianpwk/28/orig
2025-12-04T12:26:06.7945905Z  * [new branch]                gh/pianpwk/29/base      -> origin/gh/pianpwk/29/base
2025-12-04T12:26:06.7946081Z  * [new branch]                gh/pianpwk/29/head      -> origin/gh/pianpwk/29/head
2025-12-04T12:26:06.7946258Z  * [new branch]                gh/pianpwk/29/orig      -> origin/gh/pianpwk/29/orig
2025-12-04T12:26:06.7946438Z  * [new branch]                gh/pianpwk/30/base      -> origin/gh/pianpwk/30/base
2025-12-04T12:26:06.7946615Z  * [new branch]                gh/pianpwk/30/head      -> origin/gh/pianpwk/30/head
2025-12-04T12:26:06.7946843Z  * [new branch]                gh/pianpwk/30/orig      -> origin/gh/pianpwk/30/orig
2025-12-04T12:26:06.7947068Z  * [new branch]                gh/pianpwk/31/base      -> origin/gh/pianpwk/31/base
2025-12-04T12:26:06.7947245Z  * [new branch]                gh/pianpwk/31/head      -> origin/gh/pianpwk/31/head
2025-12-04T12:26:06.7947471Z  * [new branch]                gh/pianpwk/31/orig      -> origin/gh/pianpwk/31/orig
2025-12-04T12:26:06.7947649Z  * [new branch]                gh/pianpwk/32/base      -> origin/gh/pianpwk/32/base
2025-12-04T12:26:06.7947825Z  * [new branch]                gh/pianpwk/32/head      -> origin/gh/pianpwk/32/head
2025-12-04T12:26:06.7948004Z  * [new branch]                gh/pianpwk/32/orig      -> origin/gh/pianpwk/32/orig
2025-12-04T12:26:06.7948179Z  * [new branch]                gh/pianpwk/33/base      -> origin/gh/pianpwk/33/base
2025-12-04T12:26:06.7948357Z  * [new branch]                gh/pianpwk/33/head      -> origin/gh/pianpwk/33/head
2025-12-04T12:26:06.7948535Z  * [new branch]                gh/pianpwk/33/orig      -> origin/gh/pianpwk/33/orig
2025-12-04T12:26:06.7948713Z  * [new branch]                gh/pianpwk/34/base      -> origin/gh/pianpwk/34/base
2025-12-04T12:26:06.7948893Z  * [new branch]                gh/pianpwk/34/head      -> origin/gh/pianpwk/34/head
2025-12-04T12:26:06.7949074Z  * [new branch]                gh/pianpwk/34/orig      -> origin/gh/pianpwk/34/orig
2025-12-04T12:26:06.7949252Z  * [new branch]                gh/pianpwk/35/base      -> origin/gh/pianpwk/35/base
2025-12-04T12:26:06.7949430Z  * [new branch]                gh/pianpwk/35/head      -> origin/gh/pianpwk/35/head
2025-12-04T12:26:06.7949607Z  * [new branch]                gh/pianpwk/35/orig      -> origin/gh/pianpwk/35/orig
2025-12-04T12:26:06.7949783Z  * [new branch]                gh/rec/141/base         -> origin/gh/rec/141/base
2025-12-04T12:26:06.7949956Z  * [new branch]                gh/rec/141/head         -> origin/gh/rec/141/head
2025-12-04T12:26:06.7950123Z  * [new branch]                gh/rec/153/base         -> origin/gh/rec/153/base
2025-12-04T12:26:06.7950289Z  * [new branch]                gh/rec/153/head         -> origin/gh/rec/153/head
2025-12-04T12:26:06.7950456Z  * [new branch]                gh/rec/153/orig         -> origin/gh/rec/153/orig
2025-12-04T12:26:06.7950623Z  * [new branch]                gh/rec/154/base         -> origin/gh/rec/154/base
2025-12-04T12:26:06.7950792Z  * [new branch]                gh/rec/154/head         -> origin/gh/rec/154/head
2025-12-04T12:26:06.7950962Z  * [new branch]                gh/rec/154/orig         -> origin/gh/rec/154/orig
2025-12-04T12:26:06.7951130Z  * [new branch]                gh/rec/164/base         -> origin/gh/rec/164/base
2025-12-04T12:26:06.7951295Z  * [new branch]                gh/rec/164/head         -> origin/gh/rec/164/head
2025-12-04T12:26:06.7951464Z  * [new branch]                gh/rec/164/orig         -> origin/gh/rec/164/orig
2025-12-04T12:26:06.7951631Z  * [new branch]                gh/rec/166/base         -> origin/gh/rec/166/base
2025-12-04T12:26:06.7951795Z  * [new branch]                gh/rec/166/head         -> origin/gh/rec/166/head
2025-12-04T12:26:06.7951963Z  * [new branch]                gh/rec/166/orig         -> origin/gh/rec/166/orig
2025-12-04T12:26:06.7952131Z  * [new branch]                gh/rec/167/base         -> origin/gh/rec/167/base
2025-12-04T12:26:06.7952301Z  * [new branch]                gh/rec/167/head         -> origin/gh/rec/167/head
2025-12-04T12:26:06.7952466Z  * [new branch]                gh/rec/167/orig         -> origin/gh/rec/167/orig
2025-12-04T12:26:06.7952632Z  * [new branch]                gh/rec/168/base         -> origin/gh/rec/168/base
2025-12-04T12:26:06.7952797Z  * [new branch]                gh/rec/168/head         -> origin/gh/rec/168/head
2025-12-04T12:26:06.7952964Z  * [new branch]                gh/rec/168/orig         -> origin/gh/rec/168/orig
2025-12-04T12:26:06.7953128Z  * [new branch]                gh/rec/169/base         -> origin/gh/rec/169/base
2025-12-04T12:26:06.7953292Z  * [new branch]                gh/rec/169/head         -> origin/gh/rec/169/head
2025-12-04T12:26:06.7953489Z  * [new branch]                gh/rec/169/orig         -> origin/gh/rec/169/orig
2025-12-04T12:26:06.7953656Z  * [new branch]                gh/rec/170/base         -> origin/gh/rec/170/base
2025-12-04T12:26:06.7953851Z  * [new branch]                gh/rec/170/head         -> origin/gh/rec/170/head
2025-12-04T12:26:06.7954019Z  * [new branch]                gh/rec/170/orig         -> origin/gh/rec/170/orig
2025-12-04T12:26:06.7954186Z  * [new branch]                gh/rec/171/base         -> origin/gh/rec/171/base
2025-12-04T12:26:06.7954349Z  * [new branch]                gh/rec/171/head         -> origin/gh/rec/171/head
2025-12-04T12:26:06.7954514Z  * [new branch]                gh/rec/171/orig         -> origin/gh/rec/171/orig
2025-12-04T12:26:06.7954678Z  * [new branch]                gh/rec/172/base         -> origin/gh/rec/172/base
2025-12-04T12:26:06.7954845Z  * [new branch]                gh/rec/172/head         -> origin/gh/rec/172/head
2025-12-04T12:26:06.7955012Z  * [new branch]                gh/rec/172/orig         -> origin/gh/rec/172/orig
2025-12-04T12:26:06.7955175Z  * [new branch]                gh/rec/173/base         -> origin/gh/rec/173/base
2025-12-04T12:26:06.7955340Z  * [new branch]                gh/rec/173/head         -> origin/gh/rec/173/head
2025-12-04T12:26:06.7955510Z  * [new branch]                gh/rec/173/orig         -> origin/gh/rec/173/orig
2025-12-04T12:26:06.7955673Z  * [new branch]                gh/rec/174/base         -> origin/gh/rec/174/base
2025-12-04T12:26:06.7955842Z  * [new branch]                gh/rec/174/head         -> origin/gh/rec/174/head
2025-12-04T12:26:06.7956008Z  * [new branch]                gh/rec/174/orig         -> origin/gh/rec/174/orig
2025-12-04T12:26:06.7956175Z  * [new branch]                gh/rec/175/base         -> origin/gh/rec/175/base
2025-12-04T12:26:06.7956341Z  * [new branch]                gh/rec/175/head         -> origin/gh/rec/175/head
2025-12-04T12:26:06.7956505Z  * [new branch]                gh/rec/175/orig         -> origin/gh/rec/175/orig
2025-12-04T12:26:06.7956672Z  * [new branch]                gh/rec/176/base         -> origin/gh/rec/176/base
2025-12-04T12:26:06.7956872Z  * [new branch]                gh/rec/176/head         -> origin/gh/rec/176/head
2025-12-04T12:26:06.7957042Z  * [new branch]                gh/rec/176/orig         -> origin/gh/rec/176/orig
2025-12-04T12:26:06.7957209Z  * [new branch]                gh/rec/177/base         -> origin/gh/rec/177/base
2025-12-04T12:26:06.7957376Z  * [new branch]                gh/rec/177/head         -> origin/gh/rec/177/head
2025-12-04T12:26:06.7957539Z  * [new branch]                gh/rec/177/orig         -> origin/gh/rec/177/orig
2025-12-04T12:26:06.7957705Z  * [new branch]                gh/rec/178/base         -> origin/gh/rec/178/base
2025-12-04T12:26:06.7957872Z  * [new branch]                gh/rec/178/head         -> origin/gh/rec/178/head
2025-12-04T12:26:06.7958036Z  * [new branch]                gh/rec/178/orig         -> origin/gh/rec/178/orig
2025-12-04T12:26:06.7958231Z  * [new branch]                gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base
2025-12-04T12:26:06.7958443Z  * [new branch]                gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head
2025-12-04T12:26:06.7958653Z  * [new branch]                gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig
2025-12-04T12:26:06.7958857Z  * [new branch]                gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base
2025-12-04T12:26:06.7959060Z  * [new branch]                gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head
2025-12-04T12:26:06.7959264Z  * [new branch]                gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig
2025-12-04T12:26:06.7959470Z  * [new branch]                gh/robert-hardwick/5/base -> origin/gh/robert-hardwick/5/base
2025-12-04T12:26:06.7959671Z  * [new branch]                gh/robert-hardwick/5/head -> origin/gh/robert-hardwick/5/head
2025-12-04T12:26:06.7959933Z  * [new branch]                gh/robert-hardwick/5/orig -> origin/gh/robert-hardwick/5/orig
2025-12-04T12:26:06.7960138Z  * [new branch]                gh/robert-hardwick/6/base -> origin/gh/robert-hardwick/6/base
2025-12-04T12:26:06.7960374Z  * [new branch]                gh/robert-hardwick/6/head -> origin/gh/robert-hardwick/6/head
2025-12-04T12:26:06.7960575Z  * [new branch]                gh/robert-hardwick/6/orig -> origin/gh/robert-hardwick/6/orig
2025-12-04T12:26:06.7960782Z  * [new branch]                gh/robert-hardwick/7/base -> origin/gh/robert-hardwick/7/base
2025-12-04T12:26:06.7960983Z  * [new branch]                gh/robert-hardwick/7/head -> origin/gh/robert-hardwick/7/head
2025-12-04T12:26:06.7961186Z  * [new branch]                gh/robert-hardwick/7/orig -> origin/gh/robert-hardwick/7/orig
2025-12-04T12:26:06.7961389Z  * [new branch]                gh/robert-hardwick/8/base -> origin/gh/robert-hardwick/8/base
2025-12-04T12:26:06.7961594Z  * [new branch]                gh/robert-hardwick/8/head -> origin/gh/robert-hardwick/8/head
2025-12-04T12:26:06.7961798Z  * [new branch]                gh/robert-hardwick/8/orig -> origin/gh/robert-hardwick/8/orig
2025-12-04T12:26:06.7962003Z  * [new branch]                gh/robert-hardwick/9/base -> origin/gh/robert-hardwick/9/base
2025-12-04T12:26:06.7962209Z  * [new branch]                gh/robert-hardwick/9/head -> origin/gh/robert-hardwick/9/head
2025-12-04T12:26:06.7962414Z  * [new branch]                gh/robert-hardwick/9/orig -> origin/gh/robert-hardwick/9/orig
2025-12-04T12:26:06.7962605Z  * [new branch]                gh/rtimpe/1/base        -> origin/gh/rtimpe/1/base
2025-12-04T12:26:06.7962783Z  * [new branch]                gh/rtimpe/1/head        -> origin/gh/rtimpe/1/head
2025-12-04T12:26:06.7962958Z  * [new branch]                gh/rtimpe/2/base        -> origin/gh/rtimpe/2/base
2025-12-04T12:26:06.7963129Z  * [new branch]                gh/rtimpe/2/head        -> origin/gh/rtimpe/2/head
2025-12-04T12:26:06.7963304Z  * [new branch]                gh/rtimpe/22/base       -> origin/gh/rtimpe/22/base
2025-12-04T12:26:06.7963481Z  * [new branch]                gh/rtimpe/22/head       -> origin/gh/rtimpe/22/head
2025-12-04T12:26:06.7963657Z  * [new branch]                gh/rtimpe/22/orig       -> origin/gh/rtimpe/22/orig
2025-12-04T12:26:06.7963832Z  * [new branch]                gh/rtimpe/23/base       -> origin/gh/rtimpe/23/base
2025-12-04T12:26:06.7964007Z  * [new branch]                gh/rtimpe/23/head       -> origin/gh/rtimpe/23/head
2025-12-04T12:26:06.7964176Z  * [new branch]                gh/rtimpe/23/orig       -> origin/gh/rtimpe/23/orig
2025-12-04T12:26:06.7964349Z  * [new branch]                gh/rtimpe/24/base       -> origin/gh/rtimpe/24/base
2025-12-04T12:26:06.7964524Z  * [new branch]                gh/rtimpe/24/head       -> origin/gh/rtimpe/24/head
2025-12-04T12:26:06.7964694Z  * [new branch]                gh/rtimpe/24/orig       -> origin/gh/rtimpe/24/orig
2025-12-04T12:26:06.7964869Z  * [new branch]                gh/rtimpe/25/base       -> origin/gh/rtimpe/25/base
2025-12-04T12:26:06.7965041Z  * [new branch]                gh/rtimpe/25/head       -> origin/gh/rtimpe/25/head
2025-12-04T12:26:06.7965213Z  * [new branch]                gh/rtimpe/25/orig       -> origin/gh/rtimpe/25/orig
2025-12-04T12:26:06.7965392Z  * [new branch]                gh/rtimpe/26/base       -> origin/gh/rtimpe/26/base
2025-12-04T12:26:06.7965566Z  * [new branch]                gh/rtimpe/26/head       -> origin/gh/rtimpe/26/head
2025-12-04T12:26:06.7965737Z  * [new branch]                gh/rtimpe/26/orig       -> origin/gh/rtimpe/26/orig
2025-12-04T12:26:06.7965912Z  * [new branch]                gh/rtimpe/27/base       -> origin/gh/rtimpe/27/base
2025-12-04T12:26:06.7966085Z  * [new branch]                gh/rtimpe/27/head       -> origin/gh/rtimpe/27/head
2025-12-04T12:26:06.7966259Z  * [new branch]                gh/rtimpe/27/orig       -> origin/gh/rtimpe/27/orig
2025-12-04T12:26:06.7966462Z  * [new branch]                gh/rtimpe/28/base       -> origin/gh/rtimpe/28/base
2025-12-04T12:26:06.7966635Z  * [new branch]                gh/rtimpe/28/head       -> origin/gh/rtimpe/28/head
2025-12-04T12:26:06.7966850Z  * [new branch]                gh/rtimpe/28/orig       -> origin/gh/rtimpe/28/orig
2025-12-04T12:26:06.7967062Z  * [new branch]                gh/rtimpe/29/base       -> origin/gh/rtimpe/29/base
2025-12-04T12:26:06.7967236Z  * [new branch]                gh/rtimpe/29/head       -> origin/gh/rtimpe/29/head
2025-12-04T12:26:06.7967409Z  * [new branch]                gh/rtimpe/29/orig       -> origin/gh/rtimpe/29/orig
2025-12-04T12:26:06.7967582Z  * [new branch]                gh/rtimpe/3/base        -> origin/gh/rtimpe/3/base
2025-12-04T12:26:06.7967755Z  * [new branch]                gh/rtimpe/3/head        -> origin/gh/rtimpe/3/head
2025-12-04T12:26:06.7967931Z  * [new branch]                gh/rtimpe/30/base       -> origin/gh/rtimpe/30/base
2025-12-04T12:26:06.7968103Z  * [new branch]                gh/rtimpe/30/head       -> origin/gh/rtimpe/30/head
2025-12-04T12:26:06.7968274Z  * [new branch]                gh/rtimpe/30/orig       -> origin/gh/rtimpe/30/orig
2025-12-04T12:26:06.7968451Z  * [new branch]                gh/rtimpe/31/base       -> origin/gh/rtimpe/31/base
2025-12-04T12:26:06.7968626Z  * [new branch]                gh/rtimpe/31/head       -> origin/gh/rtimpe/31/head
2025-12-04T12:26:06.7968798Z  * [new branch]                gh/rtimpe/31/orig       -> origin/gh/rtimpe/31/orig
2025-12-04T12:26:06.7968971Z  * [new branch]                gh/rtimpe/32/base       -> origin/gh/rtimpe/32/base
2025-12-04T12:26:06.7969144Z  * [new branch]                gh/rtimpe/32/head       -> origin/gh/rtimpe/32/head
2025-12-04T12:26:06.7969317Z  * [new branch]                gh/rtimpe/32/orig       -> origin/gh/rtimpe/32/orig
2025-12-04T12:26:06.7969496Z  * [new branch]                gh/rtimpe/33/base       -> origin/gh/rtimpe/33/base
2025-12-04T12:26:06.7969671Z  * [new branch]                gh/rtimpe/33/head       -> origin/gh/rtimpe/33/head
2025-12-04T12:26:06.7969848Z  * [new branch]                gh/rtimpe/33/orig       -> origin/gh/rtimpe/33/orig
2025-12-04T12:26:06.7970024Z  * [new branch]                gh/rtimpe/34/base       -> origin/gh/rtimpe/34/base
2025-12-04T12:26:06.7970207Z  * [new branch]                gh/rtimpe/34/head       -> origin/gh/rtimpe/34/head
2025-12-04T12:26:06.7970383Z  * [new branch]                gh/rtimpe/34/orig       -> origin/gh/rtimpe/34/orig
2025-12-04T12:26:06.7970568Z  * [new branch]                gh/rtimpe/35/base       -> origin/gh/rtimpe/35/base
2025-12-04T12:26:06.7970742Z  * [new branch]                gh/rtimpe/35/head       -> origin/gh/rtimpe/35/head
2025-12-04T12:26:06.7970912Z  * [new branch]                gh/rtimpe/35/orig       -> origin/gh/rtimpe/35/orig
2025-12-04T12:26:06.7971083Z  * [new branch]                gh/rtimpe/4/base        -> origin/gh/rtimpe/4/base
2025-12-04T12:26:06.7971253Z  * [new branch]                gh/rtimpe/4/head        -> origin/gh/rtimpe/4/head
2025-12-04T12:26:06.7971442Z  * [new branch]                gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base
2025-12-04T12:26:06.7971641Z  * [new branch]                gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head
2025-12-04T12:26:06.7971842Z  * [new branch]                gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig
2025-12-04T12:26:06.7972042Z  * [new branch]                gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base
2025-12-04T12:26:06.7972239Z  * [new branch]                gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head
2025-12-04T12:26:06.7972429Z  * [new branch]                gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig
2025-12-04T12:26:06.7972620Z  * [new branch]                gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base
2025-12-04T12:26:06.7972808Z  * [new branch]                gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head
2025-12-04T12:26:06.7973004Z  * [new branch]                gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig
2025-12-04T12:26:06.7973248Z  * [new branch]                gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base
2025-12-04T12:26:06.7973444Z  * [new branch]                gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head
2025-12-04T12:26:06.7973665Z  * [new branch]                gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig
2025-12-04T12:26:06.7973856Z  * [new branch]                gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base
2025-12-04T12:26:06.7974047Z  * [new branch]                gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head
2025-12-04T12:26:06.7974243Z  * [new branch]                gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig
2025-12-04T12:26:06.7974437Z  * [new branch]                gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base
2025-12-04T12:26:06.7974629Z  * [new branch]                gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head
2025-12-04T12:26:06.7974842Z  * [new branch]                gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig
2025-12-04T12:26:06.7975032Z  * [new branch]                gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base
2025-12-04T12:26:06.7975223Z  * [new branch]                gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head
2025-12-04T12:26:06.7975417Z  * [new branch]                gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig
2025-12-04T12:26:06.7975609Z  * [new branch]                gh/seemethere/52/base   -> origin/gh/seemethere/52/base
2025-12-04T12:26:06.7975800Z  * [new branch]                gh/seemethere/52/head   -> origin/gh/seemethere/52/head
2025-12-04T12:26:06.7975990Z  * [new branch]                gh/seemethere/52/orig   -> origin/gh/seemethere/52/orig
2025-12-04T12:26:06.7976178Z  * [new branch]                gh/seemethere/53/base   -> origin/gh/seemethere/53/base
2025-12-04T12:26:06.7976367Z  * [new branch]                gh/seemethere/53/head   -> origin/gh/seemethere/53/head
2025-12-04T12:26:06.7976557Z  * [new branch]                gh/seemethere/53/orig   -> origin/gh/seemethere/53/orig
2025-12-04T12:26:06.7976784Z  * [new branch]                gh/seemethere/54/base   -> origin/gh/seemethere/54/base
2025-12-04T12:26:06.7976972Z  * [new branch]                gh/seemethere/54/head   -> origin/gh/seemethere/54/head
2025-12-04T12:26:06.7977164Z  * [new branch]                gh/seemethere/54/orig   -> origin/gh/seemethere/54/orig
2025-12-04T12:26:06.7977358Z  * [new branch]                gh/seemethere/55/base   -> origin/gh/seemethere/55/base
2025-12-04T12:26:06.7977550Z  * [new branch]                gh/seemethere/55/head   -> origin/gh/seemethere/55/head
2025-12-04T12:26:06.7977740Z  * [new branch]                gh/seemethere/55/orig   -> origin/gh/seemethere/55/orig
2025-12-04T12:26:06.7977933Z  * [new branch]                gh/seemethere/59/base   -> origin/gh/seemethere/59/base
2025-12-04T12:26:06.7978124Z  * [new branch]                gh/seemethere/59/head   -> origin/gh/seemethere/59/head
2025-12-04T12:26:06.7978321Z  * [new branch]                gh/seemethere/59/orig   -> origin/gh/seemethere/59/orig
2025-12-04T12:26:06.7978510Z  * [new branch]                gh/seemethere/62/base   -> origin/gh/seemethere/62/base
2025-12-04T12:26:06.7978695Z  * [new branch]                gh/seemethere/62/head   -> origin/gh/seemethere/62/head
2025-12-04T12:26:06.7978887Z  * [new branch]                gh/seemethere/62/orig   -> origin/gh/seemethere/62/orig
2025-12-04T12:26:06.7979081Z  * [new branch]                gh/seemethere/63/base   -> origin/gh/seemethere/63/base
2025-12-04T12:26:06.7979270Z  * [new branch]                gh/seemethere/63/head   -> origin/gh/seemethere/63/head
2025-12-04T12:26:06.7979463Z  * [new branch]                gh/seemethere/63/orig   -> origin/gh/seemethere/63/orig
2025-12-04T12:26:06.7979657Z  * [new branch]                gh/seemethere/71/base   -> origin/gh/seemethere/71/base
2025-12-04T12:26:06.7979851Z  * [new branch]                gh/seemethere/71/head   -> origin/gh/seemethere/71/head
2025-12-04T12:26:06.7980085Z  * [new branch]                gh/seemethere/71/orig   -> origin/gh/seemethere/71/orig
2025-12-04T12:26:06.7980269Z  * [new branch]                gh/seemethere/72/base   -> origin/gh/seemethere/72/base
2025-12-04T12:26:06.7980510Z  * [new branch]                gh/seemethere/72/head   -> origin/gh/seemethere/72/head
2025-12-04T12:26:06.7980704Z  * [new branch]                gh/seemethere/72/orig   -> origin/gh/seemethere/72/orig
2025-12-04T12:26:06.7980893Z  * [new branch]                gh/seemethere/73/base   -> origin/gh/seemethere/73/base
2025-12-04T12:26:06.7981085Z  * [new branch]                gh/seemethere/73/head   -> origin/gh/seemethere/73/head
2025-12-04T12:26:06.7981280Z  * [new branch]                gh/seemethere/73/orig   -> origin/gh/seemethere/73/orig
2025-12-04T12:26:06.7981486Z  * [new branch]                gh/seemethere/74/base   -> origin/gh/seemethere/74/base
2025-12-04T12:26:06.7981677Z  * [new branch]                gh/seemethere/74/head   -> origin/gh/seemethere/74/head
2025-12-04T12:26:06.7981869Z  * [new branch]                gh/seemethere/74/orig   -> origin/gh/seemethere/74/orig
2025-12-04T12:26:06.7982062Z  * [new branch]                gh/seemethere/75/base   -> origin/gh/seemethere/75/base
2025-12-04T12:26:06.7982255Z  * [new branch]                gh/seemethere/75/head   -> origin/gh/seemethere/75/head
2025-12-04T12:26:06.7982451Z  * [new branch]                gh/seemethere/75/orig   -> origin/gh/seemethere/75/orig
2025-12-04T12:26:06.7982643Z  * [new branch]                gh/seemethere/76/base   -> origin/gh/seemethere/76/base
2025-12-04T12:26:06.7982838Z  * [new branch]                gh/seemethere/76/head   -> origin/gh/seemethere/76/head
2025-12-04T12:26:06.7983022Z  * [new branch]                gh/seemethere/76/orig   -> origin/gh/seemethere/76/orig
2025-12-04T12:26:06.7983210Z  * [new branch]                gh/shunting314/145/base -> origin/gh/shunting314/145/base
2025-12-04T12:26:06.7983408Z  * [new branch]                gh/shunting314/145/head -> origin/gh/shunting314/145/head
2025-12-04T12:26:06.7983596Z  * [new branch]                gh/shunting314/145/orig -> origin/gh/shunting314/145/orig
2025-12-04T12:26:06.7983790Z  * [new branch]                gh/shunting314/176/base -> origin/gh/shunting314/176/base
2025-12-04T12:26:06.7983980Z  * [new branch]                gh/shunting314/176/head -> origin/gh/shunting314/176/head
2025-12-04T12:26:06.7984168Z  * [new branch]                gh/shunting314/176/orig -> origin/gh/shunting314/176/orig
2025-12-04T12:26:06.7984357Z  * [new branch]                gh/shunting314/249/base -> origin/gh/shunting314/249/base
2025-12-04T12:26:06.7984545Z  * [new branch]                gh/shunting314/249/head -> origin/gh/shunting314/249/head
2025-12-04T12:26:06.7984730Z  * [new branch]                gh/shunting314/249/orig -> origin/gh/shunting314/249/orig
2025-12-04T12:26:06.7984917Z  * [new branch]                gh/shunting314/253/base -> origin/gh/shunting314/253/base
2025-12-04T12:26:06.7985104Z  * [new branch]                gh/shunting314/253/head -> origin/gh/shunting314/253/head
2025-12-04T12:26:06.7985290Z  * [new branch]                gh/shunting314/253/orig -> origin/gh/shunting314/253/orig
2025-12-04T12:26:06.7985481Z  * [new branch]                gh/shunting314/256/base -> origin/gh/shunting314/256/base
2025-12-04T12:26:06.7985669Z  * [new branch]                gh/shunting314/256/head -> origin/gh/shunting314/256/head
2025-12-04T12:26:06.7985855Z  * [new branch]                gh/shunting314/256/orig -> origin/gh/shunting314/256/orig
2025-12-04T12:26:06.7986040Z  * [new branch]                gh/shunting314/257/base -> origin/gh/shunting314/257/base
2025-12-04T12:26:06.7986228Z  * [new branch]                gh/shunting314/257/head -> origin/gh/shunting314/257/head
2025-12-04T12:26:06.7986414Z  * [new branch]                gh/shunting314/257/orig -> origin/gh/shunting314/257/orig
2025-12-04T12:26:06.7986600Z  * [new branch]                gh/shunting314/258/base -> origin/gh/shunting314/258/base
2025-12-04T12:26:06.7986866Z  * [new branch]                gh/shunting314/258/head -> origin/gh/shunting314/258/head
2025-12-04T12:26:06.7987055Z  * [new branch]                gh/shunting314/258/orig -> origin/gh/shunting314/258/orig
2025-12-04T12:26:06.7987286Z  * [new branch]                gh/shunting314/259/base -> origin/gh/shunting314/259/base
2025-12-04T12:26:06.7987474Z  * [new branch]                gh/shunting314/259/head -> origin/gh/shunting314/259/head
2025-12-04T12:26:06.7987658Z  * [new branch]                gh/shunting314/259/orig -> origin/gh/shunting314/259/orig
2025-12-04T12:26:06.7987845Z  * [new branch]                gh/shunting314/260/base -> origin/gh/shunting314/260/base
2025-12-04T12:26:06.7988034Z  * [new branch]                gh/shunting314/260/head -> origin/gh/shunting314/260/head
2025-12-04T12:26:06.7988222Z  * [new branch]                gh/shunting314/260/orig -> origin/gh/shunting314/260/orig
2025-12-04T12:26:06.7988415Z  * [new branch]                gh/shunting314/261/base -> origin/gh/shunting314/261/base
2025-12-04T12:26:06.7988607Z  * [new branch]                gh/shunting314/261/head -> origin/gh/shunting314/261/head
2025-12-04T12:26:06.7988793Z  * [new branch]                gh/shunting314/261/orig -> origin/gh/shunting314/261/orig
2025-12-04T12:26:06.7988986Z  * [new branch]                gh/shunting314/262/base -> origin/gh/shunting314/262/base
2025-12-04T12:26:06.7989175Z  * [new branch]                gh/shunting314/262/head -> origin/gh/shunting314/262/head
2025-12-04T12:26:06.7989363Z  * [new branch]                gh/shunting314/262/orig -> origin/gh/shunting314/262/orig
2025-12-04T12:26:06.7989551Z  * [new branch]                gh/shunting314/263/base -> origin/gh/shunting314/263/base
2025-12-04T12:26:06.7989738Z  * [new branch]                gh/shunting314/263/head -> origin/gh/shunting314/263/head
2025-12-04T12:26:06.7989930Z  * [new branch]                gh/shunting314/263/orig -> origin/gh/shunting314/263/orig
2025-12-04T12:26:06.7990119Z  * [new branch]                gh/shunting314/264/base -> origin/gh/shunting314/264/base
2025-12-04T12:26:06.7990304Z  * [new branch]                gh/shunting314/264/head -> origin/gh/shunting314/264/head
2025-12-04T12:26:06.7990499Z  * [new branch]                gh/shunting314/264/orig -> origin/gh/shunting314/264/orig
2025-12-04T12:26:06.7990688Z  * [new branch]                gh/shunting314/265/base -> origin/gh/shunting314/265/base
2025-12-04T12:26:06.7990877Z  * [new branch]                gh/shunting314/265/head -> origin/gh/shunting314/265/head
2025-12-04T12:26:06.7991068Z  * [new branch]                gh/shunting314/265/orig -> origin/gh/shunting314/265/orig
2025-12-04T12:26:06.7991256Z  * [new branch]                gh/shunting314/266/base -> origin/gh/shunting314/266/base
2025-12-04T12:26:06.7991440Z  * [new branch]                gh/shunting314/266/head -> origin/gh/shunting314/266/head
2025-12-04T12:26:06.7991627Z  * [new branch]                gh/shunting314/266/orig -> origin/gh/shunting314/266/orig
2025-12-04T12:26:06.7991818Z  * [new branch]                gh/shunting314/267/base -> origin/gh/shunting314/267/base
2025-12-04T12:26:06.7992004Z  * [new branch]                gh/shunting314/267/head -> origin/gh/shunting314/267/head
2025-12-04T12:26:06.7992195Z  * [new branch]                gh/shunting314/267/orig -> origin/gh/shunting314/267/orig
2025-12-04T12:26:06.7992382Z  * [new branch]                gh/shunting314/268/base -> origin/gh/shunting314/268/base
2025-12-04T12:26:06.7992569Z  * [new branch]                gh/shunting314/268/head -> origin/gh/shunting314/268/head
2025-12-04T12:26:06.7992756Z  * [new branch]                gh/shunting314/268/orig -> origin/gh/shunting314/268/orig
2025-12-04T12:26:06.7992943Z  * [new branch]                gh/shunting314/269/base -> origin/gh/shunting314/269/base
2025-12-04T12:26:06.7993128Z  * [new branch]                gh/shunting314/269/head -> origin/gh/shunting314/269/head
2025-12-04T12:26:06.7993355Z  * [new branch]                gh/shunting314/269/orig -> origin/gh/shunting314/269/orig
2025-12-04T12:26:06.7993542Z  * [new branch]                gh/silverguo/1/base     -> origin/gh/silverguo/1/base
2025-12-04T12:26:06.7993728Z  * [new branch]                gh/silverguo/1/head     -> origin/gh/silverguo/1/head
2025-12-04T12:26:06.7993945Z  * [new branch]                gh/silverguo/2/base     -> origin/gh/silverguo/2/base
2025-12-04T12:26:06.7994128Z  * [new branch]                gh/silverguo/2/head     -> origin/gh/silverguo/2/head
2025-12-04T12:26:06.7994309Z  * [new branch]                gh/silverguo/3/base     -> origin/gh/silverguo/3/base
2025-12-04T12:26:06.7994489Z  * [new branch]                gh/silverguo/3/head     -> origin/gh/silverguo/3/head
2025-12-04T12:26:06.7994669Z  * [new branch]                gh/silverguo/4/base     -> origin/gh/silverguo/4/base
2025-12-04T12:26:06.7994851Z  * [new branch]                gh/silverguo/4/head     -> origin/gh/silverguo/4/head
2025-12-04T12:26:06.7995035Z  * [new branch]                gh/slayton58/39/base    -> origin/gh/slayton58/39/base
2025-12-04T12:26:06.7995221Z  * [new branch]                gh/slayton58/39/head    -> origin/gh/slayton58/39/head
2025-12-04T12:26:06.7995404Z  * [new branch]                gh/slayton58/39/orig    -> origin/gh/slayton58/39/orig
2025-12-04T12:26:06.7995588Z  * [new branch]                gh/slayton58/42/base    -> origin/gh/slayton58/42/base
2025-12-04T12:26:06.7995770Z  * [new branch]                gh/slayton58/42/head    -> origin/gh/slayton58/42/head
2025-12-04T12:26:06.7995953Z  * [new branch]                gh/slayton58/42/orig    -> origin/gh/slayton58/42/orig
2025-12-04T12:26:06.7996135Z  * [new branch]                gh/slayton58/43/base    -> origin/gh/slayton58/43/base
2025-12-04T12:26:06.7996313Z  * [new branch]                gh/slayton58/43/head    -> origin/gh/slayton58/43/head
2025-12-04T12:26:06.7996495Z  * [new branch]                gh/slayton58/43/orig    -> origin/gh/slayton58/43/orig
2025-12-04T12:26:06.7996571Z  * [new branch]                gh/slayton58/44/base    -> origin/gh/slayton58/44/base
2025-12-04T12:26:06.7996645Z  * [new branch]                gh/slayton58/44/head    -> origin/gh/slayton58/44/head
2025-12-04T12:26:06.7996716Z  * [new branch]                gh/slayton58/44/orig    -> origin/gh/slayton58/44/orig
2025-12-04T12:26:06.7996823Z  * [new branch]                gh/slayton58/45/base    -> origin/gh/slayton58/45/base
2025-12-04T12:26:06.7996898Z  * [new branch]                gh/slayton58/45/head    -> origin/gh/slayton58/45/head
2025-12-04T12:26:06.7996969Z  * [new branch]                gh/slayton58/45/orig    -> origin/gh/slayton58/45/orig
2025-12-04T12:26:06.7997039Z  * [new branch]                gh/slayton58/46/base    -> origin/gh/slayton58/46/base
2025-12-04T12:26:06.7997114Z  * [new branch]                gh/slayton58/46/head    -> origin/gh/slayton58/46/head
2025-12-04T12:26:06.7997186Z  * [new branch]                gh/slayton58/46/orig    -> origin/gh/slayton58/46/orig
2025-12-04T12:26:06.7997259Z  * [new branch]                gh/slayton58/6/base     -> origin/gh/slayton58/6/base
2025-12-04T12:26:06.7997337Z  * [new branch]                gh/slayton58/6/head     -> origin/gh/slayton58/6/head
2025-12-04T12:26:06.7997408Z  * [new branch]                gh/slayton58/7/base     -> origin/gh/slayton58/7/base
2025-12-04T12:26:06.7997478Z  * [new branch]                gh/slayton58/7/head     -> origin/gh/slayton58/7/head
2025-12-04T12:26:06.7997557Z  * [new branch]                gh/soulitzer/269/base   -> origin/gh/soulitzer/269/base
2025-12-04T12:26:06.7997632Z  * [new branch]                gh/soulitzer/269/head   -> origin/gh/soulitzer/269/head
2025-12-04T12:26:06.7997706Z  * [new branch]                gh/soulitzer/269/orig   -> origin/gh/soulitzer/269/orig
2025-12-04T12:26:06.7997778Z  * [new branch]                gh/soulitzer/276/base   -> origin/gh/soulitzer/276/base
2025-12-04T12:26:06.7997850Z  * [new branch]                gh/soulitzer/276/head   -> origin/gh/soulitzer/276/head
2025-12-04T12:26:06.7997963Z  * [new branch]                gh/soulitzer/276/orig   -> origin/gh/soulitzer/276/orig
2025-12-04T12:26:06.7998035Z  * [new branch]                gh/soulitzer/287/base   -> origin/gh/soulitzer/287/base
2025-12-04T12:26:06.7998109Z  * [new branch]                gh/soulitzer/287/head   -> origin/gh/soulitzer/287/head
2025-12-04T12:26:06.7998218Z  * [new branch]                gh/soulitzer/287/orig   -> origin/gh/soulitzer/287/orig
2025-12-04T12:26:06.7998291Z  * [new branch]                gh/soulitzer/296/base   -> origin/gh/soulitzer/296/base
2025-12-04T12:26:06.7998364Z  * [new branch]                gh/soulitzer/296/head   -> origin/gh/soulitzer/296/head
2025-12-04T12:26:06.7998438Z  * [new branch]                gh/soulitzer/296/orig   -> origin/gh/soulitzer/296/orig
2025-12-04T12:26:06.7998511Z  * [new branch]                gh/soulitzer/299/base   -> origin/gh/soulitzer/299/base
2025-12-04T12:26:06.7998584Z  * [new branch]                gh/soulitzer/299/head   -> origin/gh/soulitzer/299/head
2025-12-04T12:26:06.7998658Z  * [new branch]                gh/soulitzer/299/orig   -> origin/gh/soulitzer/299/orig
2025-12-04T12:26:06.7998731Z  * [new branch]                gh/soulitzer/300/base   -> origin/gh/soulitzer/300/base
2025-12-04T12:26:06.7998805Z  * [new branch]                gh/soulitzer/300/head   -> origin/gh/soulitzer/300/head
2025-12-04T12:26:06.7998879Z  * [new branch]                gh/soulitzer/300/orig   -> origin/gh/soulitzer/300/orig
2025-12-04T12:26:06.7998950Z  * [new branch]                gh/soulitzer/301/base   -> origin/gh/soulitzer/301/base
2025-12-04T12:26:06.7999021Z  * [new branch]                gh/soulitzer/301/head   -> origin/gh/soulitzer/301/head
2025-12-04T12:26:06.7999094Z  * [new branch]                gh/soulitzer/301/orig   -> origin/gh/soulitzer/301/orig
2025-12-04T12:26:06.7999166Z  * [new branch]                gh/soulitzer/313/base   -> origin/gh/soulitzer/313/base
2025-12-04T12:26:06.7999239Z  * [new branch]                gh/soulitzer/313/head   -> origin/gh/soulitzer/313/head
2025-12-04T12:26:06.7999314Z  * [new branch]                gh/soulitzer/313/orig   -> origin/gh/soulitzer/313/orig
2025-12-04T12:26:06.7999386Z  * [new branch]                gh/soulitzer/319/base   -> origin/gh/soulitzer/319/base
2025-12-04T12:26:06.7999463Z  * [new branch]                gh/soulitzer/319/head   -> origin/gh/soulitzer/319/head
2025-12-04T12:26:06.7999538Z  * [new branch]                gh/soulitzer/319/orig   -> origin/gh/soulitzer/319/orig
2025-12-04T12:26:06.7999613Z  * [new branch]                gh/soulitzer/320/base   -> origin/gh/soulitzer/320/base
2025-12-04T12:26:06.7999687Z  * [new branch]                gh/soulitzer/320/head   -> origin/gh/soulitzer/320/head
2025-12-04T12:26:06.7999760Z  * [new branch]                gh/soulitzer/320/orig   -> origin/gh/soulitzer/320/orig
2025-12-04T12:26:06.7999830Z  * [new branch]                gh/soulitzer/336/base   -> origin/gh/soulitzer/336/base
2025-12-04T12:26:06.7999903Z  * [new branch]                gh/soulitzer/336/head   -> origin/gh/soulitzer/336/head
2025-12-04T12:26:06.7999977Z  * [new branch]                gh/soulitzer/336/orig   -> origin/gh/soulitzer/336/orig
2025-12-04T12:26:06.8000048Z  * [new branch]                gh/soulitzer/347/base   -> origin/gh/soulitzer/347/base
2025-12-04T12:26:06.8000126Z  * [new branch]                gh/soulitzer/347/head   -> origin/gh/soulitzer/347/head
2025-12-04T12:26:06.8000201Z  * [new branch]                gh/soulitzer/347/orig   -> origin/gh/soulitzer/347/orig
2025-12-04T12:26:06.8000275Z  * [new branch]                gh/soulitzer/349/base   -> origin/gh/soulitzer/349/base
2025-12-04T12:26:06.8000349Z  * [new branch]                gh/soulitzer/349/head   -> origin/gh/soulitzer/349/head
2025-12-04T12:26:06.8000420Z  * [new branch]                gh/soulitzer/349/orig   -> origin/gh/soulitzer/349/orig
2025-12-04T12:26:06.8000491Z  * [new branch]                gh/soulitzer/350/base   -> origin/gh/soulitzer/350/base
2025-12-04T12:26:06.8000589Z  * [new branch]                gh/soulitzer/350/head   -> origin/gh/soulitzer/350/head
2025-12-04T12:26:06.8000662Z  * [new branch]                gh/soulitzer/350/orig   -> origin/gh/soulitzer/350/orig
2025-12-04T12:26:06.8000734Z  * [new branch]                gh/soulitzer/351/base   -> origin/gh/soulitzer/351/base
2025-12-04T12:26:06.8000846Z  * [new branch]                gh/soulitzer/351/head   -> origin/gh/soulitzer/351/head
2025-12-04T12:26:06.8000917Z  * [new branch]                gh/soulitzer/351/orig   -> origin/gh/soulitzer/351/orig
2025-12-04T12:26:06.8000991Z  * [new branch]                gh/soulitzer/353/base   -> origin/gh/soulitzer/353/base
2025-12-04T12:26:06.8001061Z  * [new branch]                gh/soulitzer/353/head   -> origin/gh/soulitzer/353/head
2025-12-04T12:26:06.8001134Z  * [new branch]                gh/soulitzer/353/orig   -> origin/gh/soulitzer/353/orig
2025-12-04T12:26:06.8001208Z  * [new branch]                gh/soulitzer/358/base   -> origin/gh/soulitzer/358/base
2025-12-04T12:26:06.8001281Z  * [new branch]                gh/soulitzer/358/head   -> origin/gh/soulitzer/358/head
2025-12-04T12:26:06.8001352Z  * [new branch]                gh/soulitzer/358/orig   -> origin/gh/soulitzer/358/orig
2025-12-04T12:26:06.8001426Z  * [new branch]                gh/soulitzer/359/base   -> origin/gh/soulitzer/359/base
2025-12-04T12:26:06.8001501Z  * [new branch]                gh/soulitzer/359/head   -> origin/gh/soulitzer/359/head
2025-12-04T12:26:06.8001573Z  * [new branch]                gh/soulitzer/359/orig   -> origin/gh/soulitzer/359/orig
2025-12-04T12:26:06.8001645Z  * [new branch]                gh/soulitzer/374/base   -> origin/gh/soulitzer/374/base
2025-12-04T12:26:06.8001718Z  * [new branch]                gh/soulitzer/374/head   -> origin/gh/soulitzer/374/head
2025-12-04T12:26:06.8001789Z  * [new branch]                gh/soulitzer/374/orig   -> origin/gh/soulitzer/374/orig
2025-12-04T12:26:06.8001863Z  * [new branch]                gh/soulitzer/375/base   -> origin/gh/soulitzer/375/base
2025-12-04T12:26:06.8001936Z  * [new branch]                gh/soulitzer/375/head   -> origin/gh/soulitzer/375/head
2025-12-04T12:26:06.8002007Z  * [new branch]                gh/soulitzer/375/orig   -> origin/gh/soulitzer/375/orig
2025-12-04T12:26:06.8002083Z  * [new branch]                gh/soulitzer/380/base   -> origin/gh/soulitzer/380/base
2025-12-04T12:26:06.8002154Z  * [new branch]                gh/soulitzer/380/head   -> origin/gh/soulitzer/380/head
2025-12-04T12:26:06.8002227Z  * [new branch]                gh/soulitzer/380/orig   -> origin/gh/soulitzer/380/orig
2025-12-04T12:26:06.8002301Z  * [new branch]                gh/soulitzer/385/base   -> origin/gh/soulitzer/385/base
2025-12-04T12:26:06.8002371Z  * [new branch]                gh/soulitzer/385/head   -> origin/gh/soulitzer/385/head
2025-12-04T12:26:06.8002446Z  * [new branch]                gh/soulitzer/385/orig   -> origin/gh/soulitzer/385/orig
2025-12-04T12:26:06.8002518Z  * [new branch]                gh/soulitzer/386/base   -> origin/gh/soulitzer/386/base
2025-12-04T12:26:06.8002592Z  * [new branch]                gh/soulitzer/386/head   -> origin/gh/soulitzer/386/head
2025-12-04T12:26:06.8002665Z  * [new branch]                gh/soulitzer/386/orig   -> origin/gh/soulitzer/386/orig
2025-12-04T12:26:06.8002739Z  * [new branch]                gh/soulitzer/387/base   -> origin/gh/soulitzer/387/base
2025-12-04T12:26:06.8002812Z  * [new branch]                gh/soulitzer/387/head   -> origin/gh/soulitzer/387/head
2025-12-04T12:26:06.8002884Z  * [new branch]                gh/soulitzer/387/orig   -> origin/gh/soulitzer/387/orig
2025-12-04T12:26:06.8002956Z  * [new branch]                gh/soulitzer/388/base   -> origin/gh/soulitzer/388/base
2025-12-04T12:26:06.8003028Z  * [new branch]                gh/soulitzer/388/head   -> origin/gh/soulitzer/388/head
2025-12-04T12:26:06.8003101Z  * [new branch]                gh/soulitzer/388/orig   -> origin/gh/soulitzer/388/orig
2025-12-04T12:26:06.8003173Z  * [new branch]                gh/soulitzer/389/base   -> origin/gh/soulitzer/389/base
2025-12-04T12:26:06.8003445Z  * [new branch]                gh/soulitzer/389/head   -> origin/gh/soulitzer/389/head
2025-12-04T12:26:06.8003519Z  * [new branch]                gh/soulitzer/389/orig   -> origin/gh/soulitzer/389/orig
2025-12-04T12:26:06.8003618Z  * [new branch]                gh/soulitzer/390/base   -> origin/gh/soulitzer/390/base
2025-12-04T12:26:06.8003691Z  * [new branch]                gh/soulitzer/390/head   -> origin/gh/soulitzer/390/head
2025-12-04T12:26:06.8003767Z  * [new branch]                gh/soulitzer/390/orig   -> origin/gh/soulitzer/390/orig
2025-12-04T12:26:06.8003838Z  * [new branch]                gh/soulitzer/391/base   -> origin/gh/soulitzer/391/base
2025-12-04T12:26:06.8003911Z  * [new branch]                gh/soulitzer/391/head   -> origin/gh/soulitzer/391/head
2025-12-04T12:26:06.8003984Z  * [new branch]                gh/soulitzer/391/orig   -> origin/gh/soulitzer/391/orig
2025-12-04T12:26:06.8004058Z  * [new branch]                gh/soulitzer/392/base   -> origin/gh/soulitzer/392/base
2025-12-04T12:26:06.8004130Z  * [new branch]                gh/soulitzer/392/head   -> origin/gh/soulitzer/392/head
2025-12-04T12:26:06.8004206Z  * [new branch]                gh/soulitzer/392/orig   -> origin/gh/soulitzer/392/orig
2025-12-04T12:26:06.8004280Z  * [new branch]                gh/swolchok/728/next    -> origin/gh/swolchok/728/next
2025-12-04T12:26:06.8004352Z  * [new branch]                gh/swolchok/819/base    -> origin/gh/swolchok/819/base
2025-12-04T12:26:06.8004422Z  * [new branch]                gh/swolchok/819/head    -> origin/gh/swolchok/819/head
2025-12-04T12:26:06.8004493Z  * [new branch]                gh/swolchok/819/orig    -> origin/gh/swolchok/819/orig
2025-12-04T12:26:06.8004568Z  * [new branch]                gh/swolchok/824/base    -> origin/gh/swolchok/824/base
2025-12-04T12:26:06.8004638Z  * [new branch]                gh/swolchok/824/head    -> origin/gh/swolchok/824/head
2025-12-04T12:26:06.8004709Z  * [new branch]                gh/swolchok/824/orig    -> origin/gh/swolchok/824/orig
2025-12-04T12:26:06.8004783Z  * [new branch]                gh/swolchok/829/base    -> origin/gh/swolchok/829/base
2025-12-04T12:26:06.8004854Z  * [new branch]                gh/swolchok/829/head    -> origin/gh/swolchok/829/head
2025-12-04T12:26:06.8004926Z  * [new branch]                gh/swolchok/829/orig    -> origin/gh/swolchok/829/orig
2025-12-04T12:26:06.8004999Z  * [new branch]                gh/swolchok/839/base    -> origin/gh/swolchok/839/base
2025-12-04T12:26:06.8005070Z  * [new branch]                gh/swolchok/839/head    -> origin/gh/swolchok/839/head
2025-12-04T12:26:06.8005141Z  * [new branch]                gh/swolchok/839/orig    -> origin/gh/swolchok/839/orig
2025-12-04T12:26:06.8005215Z  * [new branch]                gh/swolchok/841/base    -> origin/gh/swolchok/841/base
2025-12-04T12:26:06.8005286Z  * [new branch]                gh/swolchok/841/head    -> origin/gh/swolchok/841/head
2025-12-04T12:26:06.8005357Z  * [new branch]                gh/swolchok/841/orig    -> origin/gh/swolchok/841/orig
2025-12-04T12:26:06.8005430Z  * [new branch]                gh/swolchok/842/base    -> origin/gh/swolchok/842/base
2025-12-04T12:26:06.8005501Z  * [new branch]                gh/swolchok/842/head    -> origin/gh/swolchok/842/head
2025-12-04T12:26:06.8005575Z  * [new branch]                gh/swolchok/842/orig    -> origin/gh/swolchok/842/orig
2025-12-04T12:26:06.8005649Z  * [new branch]                gh/swolchok/845/base    -> origin/gh/swolchok/845/base
2025-12-04T12:26:06.8005718Z  * [new branch]                gh/swolchok/845/head    -> origin/gh/swolchok/845/head
2025-12-04T12:26:06.8005790Z  * [new branch]                gh/swolchok/845/orig    -> origin/gh/swolchok/845/orig
2025-12-04T12:26:06.8005859Z  * [new branch]                gh/swolchok/848/base    -> origin/gh/swolchok/848/base
2025-12-04T12:26:06.8005930Z  * [new branch]                gh/swolchok/848/head    -> origin/gh/swolchok/848/head
2025-12-04T12:26:06.8006027Z  * [new branch]                gh/swolchok/848/orig    -> origin/gh/swolchok/848/orig
2025-12-04T12:26:06.8006100Z  * [new branch]                gh/swolchok/856/base    -> origin/gh/swolchok/856/base
2025-12-04T12:26:06.8006171Z  * [new branch]                gh/swolchok/856/head    -> origin/gh/swolchok/856/head
2025-12-04T12:26:06.8006267Z  * [new branch]                gh/swolchok/856/orig    -> origin/gh/swolchok/856/orig
2025-12-04T12:26:06.8006338Z  * [new branch]                gh/swolchok/860/base    -> origin/gh/swolchok/860/base
2025-12-04T12:26:06.8006408Z  * [new branch]                gh/swolchok/860/head    -> origin/gh/swolchok/860/head
2025-12-04T12:26:06.8006480Z  * [new branch]                gh/swolchok/860/orig    -> origin/gh/swolchok/860/orig
2025-12-04T12:26:06.8006551Z  * [new branch]                gh/swolchok/861/base    -> origin/gh/swolchok/861/base
2025-12-04T12:26:06.8006622Z  * [new branch]                gh/swolchok/861/head    -> origin/gh/swolchok/861/head
2025-12-04T12:26:06.8006694Z  * [new branch]                gh/swolchok/861/orig    -> origin/gh/swolchok/861/orig
2025-12-04T12:26:06.8006802Z  * [new branch]                gh/swolchok/862/base    -> origin/gh/swolchok/862/base
2025-12-04T12:26:06.8006874Z  * [new branch]                gh/swolchok/862/head    -> origin/gh/swolchok/862/head
2025-12-04T12:26:06.8006947Z  * [new branch]                gh/swolchok/862/orig    -> origin/gh/swolchok/862/orig
2025-12-04T12:26:06.8007018Z  * [new branch]                gh/swolchok/863/base    -> origin/gh/swolchok/863/base
2025-12-04T12:26:06.8007090Z  * [new branch]                gh/swolchok/863/head    -> origin/gh/swolchok/863/head
2025-12-04T12:26:06.8007162Z  * [new branch]                gh/swolchok/863/orig    -> origin/gh/swolchok/863/orig
2025-12-04T12:26:06.8007233Z  * [new branch]                gh/swolchok/864/base    -> origin/gh/swolchok/864/base
2025-12-04T12:26:06.8007305Z  * [new branch]                gh/swolchok/864/head    -> origin/gh/swolchok/864/head
2025-12-04T12:26:06.8007376Z  * [new branch]                gh/swolchok/864/orig    -> origin/gh/swolchok/864/orig
2025-12-04T12:26:06.8007447Z  * [new branch]                gh/swolchok/865/base    -> origin/gh/swolchok/865/base
2025-12-04T12:26:06.8007521Z  * [new branch]                gh/swolchok/865/head    -> origin/gh/swolchok/865/head
2025-12-04T12:26:06.8007592Z  * [new branch]                gh/swolchok/865/orig    -> origin/gh/swolchok/865/orig
2025-12-04T12:26:06.8007663Z  * [new branch]                gh/swolchok/866/base    -> origin/gh/swolchok/866/base
2025-12-04T12:26:06.8007735Z  * [new branch]                gh/swolchok/866/head    -> origin/gh/swolchok/866/head
2025-12-04T12:26:06.8007806Z  * [new branch]                gh/swolchok/866/orig    -> origin/gh/swolchok/866/orig
2025-12-04T12:26:06.8007877Z  * [new branch]                gh/swolchok/867/base    -> origin/gh/swolchok/867/base
2025-12-04T12:26:06.8007950Z  * [new branch]                gh/swolchok/867/head    -> origin/gh/swolchok/867/head
2025-12-04T12:26:06.8008022Z  * [new branch]                gh/swolchok/867/orig    -> origin/gh/swolchok/867/orig
2025-12-04T12:26:06.8008092Z  * [new branch]                gh/swolchok/868/base    -> origin/gh/swolchok/868/base
2025-12-04T12:26:06.8008167Z  * [new branch]                gh/swolchok/868/head    -> origin/gh/swolchok/868/head
2025-12-04T12:26:06.8008238Z  * [new branch]                gh/swolchok/868/orig    -> origin/gh/swolchok/868/orig
2025-12-04T12:26:06.8008307Z  * [new branch]                gh/swolchok/869/base    -> origin/gh/swolchok/869/base
2025-12-04T12:26:06.8008380Z  * [new branch]                gh/swolchok/869/head    -> origin/gh/swolchok/869/head
2025-12-04T12:26:06.8008450Z  * [new branch]                gh/swolchok/869/orig    -> origin/gh/swolchok/869/orig
2025-12-04T12:26:06.8008519Z  * [new branch]                gh/swolchok/870/base    -> origin/gh/swolchok/870/base
2025-12-04T12:26:06.8008592Z  * [new branch]                gh/swolchok/870/head    -> origin/gh/swolchok/870/head
2025-12-04T12:26:06.8008701Z  * [new branch]                gh/swolchok/870/orig    -> origin/gh/swolchok/870/orig
2025-12-04T12:26:06.8008772Z  * [new branch]                gh/swolchok/871/base    -> origin/gh/swolchok/871/base
2025-12-04T12:26:06.8008873Z  * [new branch]                gh/swolchok/871/head    -> origin/gh/swolchok/871/head
2025-12-04T12:26:06.8008944Z  * [new branch]                gh/swolchok/871/orig    -> origin/gh/swolchok/871/orig
2025-12-04T12:26:06.8009018Z  * [new branch]                gh/teja-rao/4/base      -> origin/gh/teja-rao/4/base
2025-12-04T12:26:06.8009088Z  * [new branch]                gh/teja-rao/4/head      -> origin/gh/teja-rao/4/head
2025-12-04T12:26:06.8009155Z  * [new branch]                gh/teja-rao/4/orig      -> origin/gh/teja-rao/4/orig
2025-12-04T12:26:06.8009230Z  * [new branch]                gh/tianyu-l/2/base      -> origin/gh/tianyu-l/2/base
2025-12-04T12:26:06.8009300Z  * [new branch]                gh/tianyu-l/2/head      -> origin/gh/tianyu-l/2/head
2025-12-04T12:26:06.8009370Z  * [new branch]                gh/tianyu-l/2/orig      -> origin/gh/tianyu-l/2/orig
2025-12-04T12:26:06.8009440Z  * [new branch]                gh/tianyu-l/3/base      -> origin/gh/tianyu-l/3/base
2025-12-04T12:26:06.8009509Z  * [new branch]                gh/tianyu-l/3/orig      -> origin/gh/tianyu-l/3/orig
2025-12-04T12:26:06.8009576Z  * [new branch]                gh/tianyu-l/4/base      -> origin/gh/tianyu-l/4/base
2025-12-04T12:26:06.8009644Z  * [new branch]                gh/tianyu-l/4/head      -> origin/gh/tianyu-l/4/head
2025-12-04T12:26:06.8009713Z  * [new branch]                gh/tianyu-l/4/orig      -> origin/gh/tianyu-l/4/orig
2025-12-04T12:26:06.8009802Z  * [new branch]                gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base
2025-12-04T12:26:06.8009890Z  * [new branch]                gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head
2025-12-04T12:26:06.8009975Z  * [new branch]                gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig
2025-12-04T12:26:06.8010059Z  * [new branch]                gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base
2025-12-04T12:26:06.8010145Z  * [new branch]                gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head
2025-12-04T12:26:06.8010229Z  * [new branch]                gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig
2025-12-04T12:26:06.8010310Z  * [new branch]                gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base
2025-12-04T12:26:06.8010393Z  * [new branch]                gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head
2025-12-04T12:26:06.8010475Z  * [new branch]                gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig
2025-12-04T12:26:06.8010562Z  * [new branch]                gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base
2025-12-04T12:26:06.8010647Z  * [new branch]                gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head
2025-12-04T12:26:06.8010729Z  * [new branch]                gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig
2025-12-04T12:26:06.8010814Z  * [new branch]                gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base
2025-12-04T12:26:06.8010897Z  * [new branch]                gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head
2025-12-04T12:26:06.8010979Z  * [new branch]                gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig
2025-12-04T12:26:06.8011065Z  * [new branch]                gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base
2025-12-04T12:26:06.8011147Z  * [new branch]                gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head
2025-12-04T12:26:06.8011229Z  * [new branch]                gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig
2025-12-04T12:26:06.8011311Z  * [new branch]                gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base
2025-12-04T12:26:06.8011446Z  * [new branch]                gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head
2025-12-04T12:26:06.8011542Z  * [new branch]                gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig
2025-12-04T12:26:06.8011653Z  * [new branch]                gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base
2025-12-04T12:26:06.8011756Z  * [new branch]                gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head
2025-12-04T12:26:06.8011838Z  * [new branch]                gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig
2025-12-04T12:26:06.8011921Z  * [new branch]                gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base
2025-12-04T12:26:06.8012002Z  * [new branch]                gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head
2025-12-04T12:26:06.8012085Z  * [new branch]                gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig
2025-12-04T12:26:06.8012180Z  * [new branch]                gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base
2025-12-04T12:26:06.8012263Z  * [new branch]                gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head
2025-12-04T12:26:06.8012345Z  * [new branch]                gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig
2025-12-04T12:26:06.8012428Z  * [new branch]                gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base
2025-12-04T12:26:06.8012510Z  * [new branch]                gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head
2025-12-04T12:26:06.8012592Z  * [new branch]                gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig
2025-12-04T12:26:06.8012674Z  * [new branch]                gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base
2025-12-04T12:26:06.8012756Z  * [new branch]                gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head
2025-12-04T12:26:06.8012839Z  * [new branch]                gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig
2025-12-04T12:26:06.8012922Z  * [new branch]                gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base
2025-12-04T12:26:06.8013004Z  * [new branch]                gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head
2025-12-04T12:26:06.8013086Z  * [new branch]                gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig
2025-12-04T12:26:06.8013170Z  * [new branch]                gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base
2025-12-04T12:26:06.8013252Z  * [new branch]                gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head
2025-12-04T12:26:06.8013334Z  * [new branch]                gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig
2025-12-04T12:26:06.8013415Z  * [new branch]                gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base
2025-12-04T12:26:06.8013496Z  * [new branch]                gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head
2025-12-04T12:26:06.8013579Z  * [new branch]                gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig
2025-12-04T12:26:06.8013661Z  * [new branch]                gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base
2025-12-04T12:26:06.8013744Z  * [new branch]                gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head
2025-12-04T12:26:06.8013827Z  * [new branch]                gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig
2025-12-04T12:26:06.8013908Z  * [new branch]                gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base
2025-12-04T12:26:06.8013990Z  * [new branch]                gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head
2025-12-04T12:26:06.8014069Z  * [new branch]                gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig
2025-12-04T12:26:06.8014151Z  * [new branch]                gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base
2025-12-04T12:26:06.8014234Z  * [new branch]                gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head
2025-12-04T12:26:06.8014340Z  * [new branch]                gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig
2025-12-04T12:26:06.8014424Z  * [new branch]                gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base
2025-12-04T12:26:06.8014530Z  * [new branch]                gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head
2025-12-04T12:26:06.8014612Z  * [new branch]                gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig
2025-12-04T12:26:06.8014694Z  * [new branch]                gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base
2025-12-04T12:26:06.8014777Z  * [new branch]                gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head
2025-12-04T12:26:06.8014859Z  * [new branch]                gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig
2025-12-04T12:26:06.8014940Z  * [new branch]                gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base
2025-12-04T12:26:06.8015024Z  * [new branch]                gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head
2025-12-04T12:26:06.8015105Z  * [new branch]                gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig
2025-12-04T12:26:06.8015187Z  * [new branch]                gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base
2025-12-04T12:26:06.8015271Z  * [new branch]                gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head
2025-12-04T12:26:06.8015353Z  * [new branch]                gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig
2025-12-04T12:26:06.8015434Z  * [new branch]                gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base
2025-12-04T12:26:06.8015514Z  * [new branch]                gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head
2025-12-04T12:26:06.8015594Z  * [new branch]                gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig
2025-12-04T12:26:06.8015676Z  * [new branch]                gh/tugsbayasgalan/70/base -> origin/gh/tugsbayasgalan/70/base
2025-12-04T12:26:06.8015760Z  * [new branch]                gh/tugsbayasgalan/70/head -> origin/gh/tugsbayasgalan/70/head
2025-12-04T12:26:06.8015842Z  * [new branch]                gh/tugsbayasgalan/70/orig -> origin/gh/tugsbayasgalan/70/orig
2025-12-04T12:26:06.8015927Z  * [new branch]                gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base
2025-12-04T12:26:06.8016008Z  * [new branch]                gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head
2025-12-04T12:26:06.8016090Z  * [new branch]                gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig
2025-12-04T12:26:06.8016173Z  * [new branch]                gh/tugsbayasgalan/72/base -> origin/gh/tugsbayasgalan/72/base
2025-12-04T12:26:06.8016255Z  * [new branch]                gh/tugsbayasgalan/72/head -> origin/gh/tugsbayasgalan/72/head
2025-12-04T12:26:06.8016337Z  * [new branch]                gh/tugsbayasgalan/72/orig -> origin/gh/tugsbayasgalan/72/orig
2025-12-04T12:26:06.8016421Z  * [new branch]                gh/tugsbayasgalan/73/base -> origin/gh/tugsbayasgalan/73/base
2025-12-04T12:26:06.8016503Z  * [new branch]                gh/tugsbayasgalan/73/head -> origin/gh/tugsbayasgalan/73/head
2025-12-04T12:26:06.8016585Z  * [new branch]                gh/tugsbayasgalan/73/orig -> origin/gh/tugsbayasgalan/73/orig
2025-12-04T12:26:06.8016669Z  * [new branch]                gh/tugsbayasgalan/74/base -> origin/gh/tugsbayasgalan/74/base
2025-12-04T12:26:06.8016787Z  * [new branch]                gh/tugsbayasgalan/74/head -> origin/gh/tugsbayasgalan/74/head
2025-12-04T12:26:06.8016872Z  * [new branch]                gh/tugsbayasgalan/74/orig -> origin/gh/tugsbayasgalan/74/orig
2025-12-04T12:26:06.8016953Z  * [new branch]                gh/tugsbayasgalan/75/base -> origin/gh/tugsbayasgalan/75/base
2025-12-04T12:26:06.8017035Z  * [new branch]                gh/tugsbayasgalan/75/head -> origin/gh/tugsbayasgalan/75/head
2025-12-04T12:26:06.8017117Z  * [new branch]                gh/tugsbayasgalan/75/orig -> origin/gh/tugsbayasgalan/75/orig
2025-12-04T12:26:06.8017233Z  * [new branch]                gh/tugsbayasgalan/76/base -> origin/gh/tugsbayasgalan/76/base
2025-12-04T12:26:06.8017316Z  * [new branch]                gh/tugsbayasgalan/76/head -> origin/gh/tugsbayasgalan/76/head
2025-12-04T12:26:06.8017431Z  * [new branch]                gh/tugsbayasgalan/76/orig -> origin/gh/tugsbayasgalan/76/orig
2025-12-04T12:26:06.8017512Z  * [new branch]                gh/tugsbayasgalan/77/base -> origin/gh/tugsbayasgalan/77/base
2025-12-04T12:26:06.8017594Z  * [new branch]                gh/tugsbayasgalan/77/head -> origin/gh/tugsbayasgalan/77/head
2025-12-04T12:26:06.8017676Z  * [new branch]                gh/tugsbayasgalan/77/orig -> origin/gh/tugsbayasgalan/77/orig
2025-12-04T12:26:06.8017760Z  * [new branch]                gh/tugsbayasgalan/78/base -> origin/gh/tugsbayasgalan/78/base
2025-12-04T12:26:06.8017842Z  * [new branch]                gh/tugsbayasgalan/78/head -> origin/gh/tugsbayasgalan/78/head
2025-12-04T12:26:06.8017925Z  * [new branch]                gh/tugsbayasgalan/78/orig -> origin/gh/tugsbayasgalan/78/orig
2025-12-04T12:26:06.8018007Z  * [new branch]                gh/tugsbayasgalan/79/base -> origin/gh/tugsbayasgalan/79/base
2025-12-04T12:26:06.8018088Z  * [new branch]                gh/tugsbayasgalan/79/head -> origin/gh/tugsbayasgalan/79/head
2025-12-04T12:26:06.8018171Z  * [new branch]                gh/tugsbayasgalan/79/orig -> origin/gh/tugsbayasgalan/79/orig
2025-12-04T12:26:06.8018252Z  * [new branch]                gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base
2025-12-04T12:26:06.8018331Z  * [new branch]                gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head
2025-12-04T12:26:06.8018411Z  * [new branch]                gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig
2025-12-04T12:26:06.8018493Z  * [new branch]                gh/tugsbayasgalan/80/base -> origin/gh/tugsbayasgalan/80/base
2025-12-04T12:26:06.8018575Z  * [new branch]                gh/tugsbayasgalan/80/head -> origin/gh/tugsbayasgalan/80/head
2025-12-04T12:26:06.8018658Z  * [new branch]                gh/tugsbayasgalan/80/orig -> origin/gh/tugsbayasgalan/80/orig
2025-12-04T12:26:06.8018739Z  * [new branch]                gh/tugsbayasgalan/81/base -> origin/gh/tugsbayasgalan/81/base
2025-12-04T12:26:06.8018824Z  * [new branch]                gh/tugsbayasgalan/81/head -> origin/gh/tugsbayasgalan/81/head
2025-12-04T12:26:06.8018905Z  * [new branch]                gh/tugsbayasgalan/81/orig -> origin/gh/tugsbayasgalan/81/orig
2025-12-04T12:26:06.8018986Z  * [new branch]                gh/tugsbayasgalan/82/base -> origin/gh/tugsbayasgalan/82/base
2025-12-04T12:26:06.8019068Z  * [new branch]                gh/tugsbayasgalan/82/head -> origin/gh/tugsbayasgalan/82/head
2025-12-04T12:26:06.8019150Z  * [new branch]                gh/tugsbayasgalan/82/orig -> origin/gh/tugsbayasgalan/82/orig
2025-12-04T12:26:06.8019232Z  * [new branch]                gh/tugsbayasgalan/83/base -> origin/gh/tugsbayasgalan/83/base
2025-12-04T12:26:06.8019316Z  * [new branch]                gh/tugsbayasgalan/83/head -> origin/gh/tugsbayasgalan/83/head
2025-12-04T12:26:06.8019397Z  * [new branch]                gh/tugsbayasgalan/83/orig -> origin/gh/tugsbayasgalan/83/orig
2025-12-04T12:26:06.8019479Z  * [new branch]                gh/tugsbayasgalan/84/base -> origin/gh/tugsbayasgalan/84/base
2025-12-04T12:26:06.8019561Z  * [new branch]                gh/tugsbayasgalan/84/head -> origin/gh/tugsbayasgalan/84/head
2025-12-04T12:26:06.8019642Z  * [new branch]                gh/tugsbayasgalan/84/orig -> origin/gh/tugsbayasgalan/84/orig
2025-12-04T12:26:06.8019723Z  * [new branch]                gh/tugsbayasgalan/85/base -> origin/gh/tugsbayasgalan/85/base
2025-12-04T12:26:06.8019806Z  * [new branch]                gh/tugsbayasgalan/85/head -> origin/gh/tugsbayasgalan/85/head
2025-12-04T12:26:06.8019887Z  * [new branch]                gh/tugsbayasgalan/85/orig -> origin/gh/tugsbayasgalan/85/orig
2025-12-04T12:26:06.8019993Z  * [new branch]                gh/tugsbayasgalan/86/base -> origin/gh/tugsbayasgalan/86/base
2025-12-04T12:26:06.8020075Z  * [new branch]                gh/tugsbayasgalan/86/head -> origin/gh/tugsbayasgalan/86/head
2025-12-04T12:26:06.8020156Z  * [new branch]                gh/tugsbayasgalan/86/orig -> origin/gh/tugsbayasgalan/86/orig
2025-12-04T12:26:06.8020262Z  * [new branch]                gh/tugsbayasgalan/87/base -> origin/gh/tugsbayasgalan/87/base
2025-12-04T12:26:06.8020344Z  * [new branch]                gh/tugsbayasgalan/87/head -> origin/gh/tugsbayasgalan/87/head
2025-12-04T12:26:06.8020425Z  * [new branch]                gh/tugsbayasgalan/87/orig -> origin/gh/tugsbayasgalan/87/orig
2025-12-04T12:26:06.8020507Z  * [new branch]                gh/tugsbayasgalan/88/base -> origin/gh/tugsbayasgalan/88/base
2025-12-04T12:26:06.8020588Z  * [new branch]                gh/tugsbayasgalan/88/head -> origin/gh/tugsbayasgalan/88/head
2025-12-04T12:26:06.8020670Z  * [new branch]                gh/tugsbayasgalan/88/orig -> origin/gh/tugsbayasgalan/88/orig
2025-12-04T12:26:06.8020754Z  * [new branch]                gh/tugsbayasgalan/89/base -> origin/gh/tugsbayasgalan/89/base
2025-12-04T12:26:06.8020835Z  * [new branch]                gh/tugsbayasgalan/89/head -> origin/gh/tugsbayasgalan/89/head
2025-12-04T12:26:06.8020918Z  * [new branch]                gh/tugsbayasgalan/89/orig -> origin/gh/tugsbayasgalan/89/orig
2025-12-04T12:26:06.8021000Z  * [new branch]                gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base
2025-12-04T12:26:06.8021080Z  * [new branch]                gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head
2025-12-04T12:26:06.8021160Z  * [new branch]                gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig
2025-12-04T12:26:06.8021242Z  * [new branch]                gh/tugsbayasgalan/90/base -> origin/gh/tugsbayasgalan/90/base
2025-12-04T12:26:06.8021323Z  * [new branch]                gh/tugsbayasgalan/90/head -> origin/gh/tugsbayasgalan/90/head
2025-12-04T12:26:06.8021406Z  * [new branch]                gh/tugsbayasgalan/90/orig -> origin/gh/tugsbayasgalan/90/orig
2025-12-04T12:26:06.8021488Z  * [new branch]                gh/tugsbayasgalan/91/base -> origin/gh/tugsbayasgalan/91/base
2025-12-04T12:26:06.8021569Z  * [new branch]                gh/tugsbayasgalan/91/head -> origin/gh/tugsbayasgalan/91/head
2025-12-04T12:26:06.8021654Z  * [new branch]                gh/tugsbayasgalan/91/orig -> origin/gh/tugsbayasgalan/91/orig
2025-12-04T12:26:06.8021736Z  * [new branch]                gh/tugsbayasgalan/92/base -> origin/gh/tugsbayasgalan/92/base
2025-12-04T12:26:06.8021817Z  * [new branch]                gh/tugsbayasgalan/92/head -> origin/gh/tugsbayasgalan/92/head
2025-12-04T12:26:06.8021899Z  * [new branch]                gh/tugsbayasgalan/92/orig -> origin/gh/tugsbayasgalan/92/orig
2025-12-04T12:26:06.8021980Z  * [new branch]                gh/tugsbayasgalan/93/base -> origin/gh/tugsbayasgalan/93/base
2025-12-04T12:26:06.8022061Z  * [new branch]                gh/tugsbayasgalan/93/head -> origin/gh/tugsbayasgalan/93/head
2025-12-04T12:26:06.8022145Z  * [new branch]                gh/tugsbayasgalan/93/orig -> origin/gh/tugsbayasgalan/93/orig
2025-12-04T12:26:06.8022212Z  * [new branch]                gh/v0i0/14/base         -> origin/gh/v0i0/14/base
2025-12-04T12:26:06.8022278Z  * [new branch]                gh/v0i0/14/head         -> origin/gh/v0i0/14/head
2025-12-04T12:26:06.8022343Z  * [new branch]                gh/v0i0/14/orig         -> origin/gh/v0i0/14/orig
2025-12-04T12:26:06.8022407Z  * [new branch]                gh/v0i0/15/base         -> origin/gh/v0i0/15/base
2025-12-04T12:26:06.8022469Z  * [new branch]                gh/v0i0/15/head         -> origin/gh/v0i0/15/head
2025-12-04T12:26:06.8022534Z  * [new branch]                gh/v0i0/15/orig         -> origin/gh/v0i0/15/orig
2025-12-04T12:26:06.8022596Z  * [new branch]                gh/v0i0/16/base         -> origin/gh/v0i0/16/base
2025-12-04T12:26:06.8022657Z  * [new branch]                gh/v0i0/16/head         -> origin/gh/v0i0/16/head
2025-12-04T12:26:06.8022756Z  * [new branch]                gh/v0i0/16/orig         -> origin/gh/v0i0/16/orig
2025-12-04T12:26:06.8022819Z  * [new branch]                gh/v0i0/17/base         -> origin/gh/v0i0/17/base
2025-12-04T12:26:06.8022901Z  * [new branch]                gh/v0i0/17/head         -> origin/gh/v0i0/17/head
2025-12-04T12:26:06.8022964Z  * [new branch]                gh/v0i0/17/orig         -> origin/gh/v0i0/17/orig
2025-12-04T12:26:06.8023025Z  * [new branch]                gh/v0i0/18/base         -> origin/gh/v0i0/18/base
2025-12-04T12:26:06.8023086Z  * [new branch]                gh/v0i0/18/head         -> origin/gh/v0i0/18/head
2025-12-04T12:26:06.8023149Z  * [new branch]                gh/v0i0/18/orig         -> origin/gh/v0i0/18/orig
2025-12-04T12:26:06.8023210Z  * [new branch]                gh/v0i0/19/base         -> origin/gh/v0i0/19/base
2025-12-04T12:26:06.8023271Z  * [new branch]                gh/v0i0/19/head         -> origin/gh/v0i0/19/head
2025-12-04T12:26:06.8023336Z  * [new branch]                gh/v0i0/19/orig         -> origin/gh/v0i0/19/orig
2025-12-04T12:26:06.8023416Z  * [new branch]                gh/vishal9-team/1/base  -> origin/gh/vishal9-team/1/base
2025-12-04T12:26:06.8023494Z  * [new branch]                gh/vishal9-team/1/head  -> origin/gh/vishal9-team/1/head
2025-12-04T12:26:06.8023571Z  * [new branch]                gh/vishal9-team/2/base  -> origin/gh/vishal9-team/2/base
2025-12-04T12:26:06.8023646Z  * [new branch]                gh/vishal9-team/2/head  -> origin/gh/vishal9-team/2/head
2025-12-04T12:26:06.8023723Z  * [new branch]                gh/vishal9-team/2/orig  -> origin/gh/vishal9-team/2/orig
2025-12-04T12:26:06.8023797Z  * [new branch]                gh/vishal9-team/3/base  -> origin/gh/vishal9-team/3/base
2025-12-04T12:26:06.8023869Z  * [new branch]                gh/vishal9-team/3/head  -> origin/gh/vishal9-team/3/head
2025-12-04T12:26:06.8023944Z  * [new branch]                gh/vishal9-team/3/orig  -> origin/gh/vishal9-team/3/orig
2025-12-04T12:26:06.8024018Z  * [new branch]                gh/vishal9-team/4/base  -> origin/gh/vishal9-team/4/base
2025-12-04T12:26:06.8024092Z  * [new branch]                gh/vishal9-team/4/head  -> origin/gh/vishal9-team/4/head
2025-12-04T12:26:06.8024168Z  * [new branch]                gh/vishal9-team/4/orig  -> origin/gh/vishal9-team/4/orig
2025-12-04T12:26:06.8024234Z  * [new branch]                gh/vkuzo/1/next         -> origin/gh/vkuzo/1/next
2025-12-04T12:26:06.8024300Z  * [new branch]                gh/vkuzo/2/next         -> origin/gh/vkuzo/2/next
2025-12-04T12:26:06.8024366Z  * [new branch]                gh/vkuzo/3/next         -> origin/gh/vkuzo/3/next
2025-12-04T12:26:06.8024439Z  * [new branch]                gh/wconstab/424/base    -> origin/gh/wconstab/424/base
2025-12-04T12:26:06.8024512Z  * [new branch]                gh/wconstab/424/head    -> origin/gh/wconstab/424/head
2025-12-04T12:26:06.8024583Z  * [new branch]                gh/wconstab/424/orig    -> origin/gh/wconstab/424/orig
2025-12-04T12:26:06.8024655Z  * [new branch]                gh/wconstab/435/base    -> origin/gh/wconstab/435/base
2025-12-04T12:26:06.8024727Z  * [new branch]                gh/wconstab/435/head    -> origin/gh/wconstab/435/head
2025-12-04T12:26:06.8024800Z  * [new branch]                gh/wconstab/435/orig    -> origin/gh/wconstab/435/orig
2025-12-04T12:26:06.8024869Z  * [new branch]                gh/wconstab/444/base    -> origin/gh/wconstab/444/base
2025-12-04T12:26:06.8024940Z  * [new branch]                gh/wconstab/444/head    -> origin/gh/wconstab/444/head
2025-12-04T12:26:06.8025011Z  * [new branch]                gh/wconstab/444/orig    -> origin/gh/wconstab/444/orig
2025-12-04T12:26:06.8025081Z  * [new branch]                gh/wconstab/447/base    -> origin/gh/wconstab/447/base
2025-12-04T12:26:06.8025153Z  * [new branch]                gh/wconstab/447/head    -> origin/gh/wconstab/447/head
2025-12-04T12:26:06.8025224Z  * [new branch]                gh/wconstab/447/orig    -> origin/gh/wconstab/447/orig
2025-12-04T12:26:06.8025317Z  * [new branch]                gh/wconstab/448/base    -> origin/gh/wconstab/448/base
2025-12-04T12:26:06.8025392Z  * [new branch]                gh/wconstab/448/head    -> origin/gh/wconstab/448/head
2025-12-04T12:26:06.8025487Z  * [new branch]                gh/wconstab/448/orig    -> origin/gh/wconstab/448/orig
2025-12-04T12:26:06.8025558Z  * [new branch]                gh/wconstab/449/base    -> origin/gh/wconstab/449/base
2025-12-04T12:26:06.8025632Z  * [new branch]                gh/wconstab/449/head    -> origin/gh/wconstab/449/head
2025-12-04T12:26:06.8025703Z  * [new branch]                gh/wconstab/449/orig    -> origin/gh/wconstab/449/orig
2025-12-04T12:26:06.8025772Z  * [new branch]                gh/wconstab/450/base    -> origin/gh/wconstab/450/base
2025-12-04T12:26:06.8025844Z  * [new branch]                gh/wconstab/450/head    -> origin/gh/wconstab/450/head
2025-12-04T12:26:06.8025913Z  * [new branch]                gh/wconstab/450/orig    -> origin/gh/wconstab/450/orig
2025-12-04T12:26:06.8025985Z  * [new branch]                gh/wconstab/451/base    -> origin/gh/wconstab/451/base
2025-12-04T12:26:06.8026056Z  * [new branch]                gh/wconstab/451/head    -> origin/gh/wconstab/451/head
2025-12-04T12:26:06.8026128Z  * [new branch]                gh/wconstab/451/orig    -> origin/gh/wconstab/451/orig
2025-12-04T12:26:06.8026197Z  * [new branch]                gh/wconstab/452/base    -> origin/gh/wconstab/452/base
2025-12-04T12:26:06.8026269Z  * [new branch]                gh/wconstab/452/head    -> origin/gh/wconstab/452/head
2025-12-04T12:26:06.8026338Z  * [new branch]                gh/wconstab/452/orig    -> origin/gh/wconstab/452/orig
2025-12-04T12:26:06.8026409Z  * [new branch]                gh/wconstab/453/base    -> origin/gh/wconstab/453/base
2025-12-04T12:26:06.8026479Z  * [new branch]                gh/wconstab/453/head    -> origin/gh/wconstab/453/head
2025-12-04T12:26:06.8026551Z  * [new branch]                gh/wconstab/453/orig    -> origin/gh/wconstab/453/orig
2025-12-04T12:26:06.8026623Z  * [new branch]                gh/wconstab/454/base    -> origin/gh/wconstab/454/base
2025-12-04T12:26:06.8026693Z  * [new branch]                gh/wconstab/454/head    -> origin/gh/wconstab/454/head
2025-12-04T12:26:06.8026803Z  * [new branch]                gh/wconstab/454/orig    -> origin/gh/wconstab/454/orig
2025-12-04T12:26:06.8026878Z  * [new branch]                gh/wconstab/455/base    -> origin/gh/wconstab/455/base
2025-12-04T12:26:06.8026952Z  * [new branch]                gh/wconstab/455/head    -> origin/gh/wconstab/455/head
2025-12-04T12:26:06.8027025Z  * [new branch]                gh/wconstab/455/orig    -> origin/gh/wconstab/455/orig
2025-12-04T12:26:06.8027098Z  * [new branch]                gh/wconstab/456/base    -> origin/gh/wconstab/456/base
2025-12-04T12:26:06.8027169Z  * [new branch]                gh/wconstab/456/head    -> origin/gh/wconstab/456/head
2025-12-04T12:26:06.8027240Z  * [new branch]                gh/wconstab/456/orig    -> origin/gh/wconstab/456/orig
2025-12-04T12:26:06.8027314Z  * [new branch]                gh/wconstab/457/base    -> origin/gh/wconstab/457/base
2025-12-04T12:26:06.8027385Z  * [new branch]                gh/wconstab/457/head    -> origin/gh/wconstab/457/head
2025-12-04T12:26:06.8027456Z  * [new branch]                gh/wconstab/457/orig    -> origin/gh/wconstab/457/orig
2025-12-04T12:26:06.8027530Z  * [new branch]                gh/wconstab/458/base    -> origin/gh/wconstab/458/base
2025-12-04T12:26:06.8027600Z  * [new branch]                gh/wconstab/458/head    -> origin/gh/wconstab/458/head
2025-12-04T12:26:06.8027670Z  * [new branch]                gh/wconstab/458/orig    -> origin/gh/wconstab/458/orig
2025-12-04T12:26:06.8027740Z  * [new branch]                gh/wconstab/459/base    -> origin/gh/wconstab/459/base
2025-12-04T12:26:06.8027810Z  * [new branch]                gh/wconstab/459/head    -> origin/gh/wconstab/459/head
2025-12-04T12:26:06.8027918Z  * [new branch]                gh/wconstab/459/orig    -> origin/gh/wconstab/459/orig
2025-12-04T12:26:06.8027989Z  * [new branch]                gh/wconstab/460/base    -> origin/gh/wconstab/460/base
2025-12-04T12:26:06.8028058Z  * [new branch]                gh/wconstab/460/head    -> origin/gh/wconstab/460/head
2025-12-04T12:26:06.8028164Z  * [new branch]                gh/wconstab/460/orig    -> origin/gh/wconstab/460/orig
2025-12-04T12:26:06.8028233Z  * [new branch]                gh/wconstab/461/base    -> origin/gh/wconstab/461/base
2025-12-04T12:26:06.8028303Z  * [new branch]                gh/wconstab/461/head    -> origin/gh/wconstab/461/head
2025-12-04T12:26:06.8028374Z  * [new branch]                gh/wconstab/461/orig    -> origin/gh/wconstab/461/orig
2025-12-04T12:26:06.8028443Z  * [new branch]                gh/wconstab/462/base    -> origin/gh/wconstab/462/base
2025-12-04T12:26:06.8028512Z  * [new branch]                gh/wconstab/462/head    -> origin/gh/wconstab/462/head
2025-12-04T12:26:06.8028585Z  * [new branch]                gh/wconstab/462/orig    -> origin/gh/wconstab/462/orig
2025-12-04T12:26:06.8028655Z  * [new branch]                gh/wconstab/463/base    -> origin/gh/wconstab/463/base
2025-12-04T12:26:06.8028725Z  * [new branch]                gh/wconstab/463/head    -> origin/gh/wconstab/463/head
2025-12-04T12:26:06.8028798Z  * [new branch]                gh/wconstab/463/orig    -> origin/gh/wconstab/463/orig
2025-12-04T12:26:06.8028868Z  * [new branch]                gh/wconstab/464/base    -> origin/gh/wconstab/464/base
2025-12-04T12:26:06.8028937Z  * [new branch]                gh/wconstab/464/head    -> origin/gh/wconstab/464/head
2025-12-04T12:26:06.8029009Z  * [new branch]                gh/wconstab/464/orig    -> origin/gh/wconstab/464/orig
2025-12-04T12:26:06.8029079Z  * [new branch]                gh/wconstab/465/base    -> origin/gh/wconstab/465/base
2025-12-04T12:26:06.8029148Z  * [new branch]                gh/wconstab/465/head    -> origin/gh/wconstab/465/head
2025-12-04T12:26:06.8029220Z  * [new branch]                gh/wconstab/465/orig    -> origin/gh/wconstab/465/orig
2025-12-04T12:26:06.8029290Z  * [new branch]                gh/wconstab/466/base    -> origin/gh/wconstab/466/base
2025-12-04T12:26:06.8029360Z  * [new branch]                gh/wconstab/466/head    -> origin/gh/wconstab/466/head
2025-12-04T12:26:06.8029432Z  * [new branch]                gh/wconstab/466/orig    -> origin/gh/wconstab/466/orig
2025-12-04T12:26:06.8029502Z  * [new branch]                gh/wconstab/467/base    -> origin/gh/wconstab/467/base
2025-12-04T12:26:06.8029572Z  * [new branch]                gh/wconstab/467/head    -> origin/gh/wconstab/467/head
2025-12-04T12:26:06.8029642Z  * [new branch]                gh/wconstab/467/orig    -> origin/gh/wconstab/467/orig
2025-12-04T12:26:06.8029712Z  * [new branch]                gh/wconstab/468/base    -> origin/gh/wconstab/468/base
2025-12-04T12:26:06.8029783Z  * [new branch]                gh/wconstab/468/head    -> origin/gh/wconstab/468/head
2025-12-04T12:26:06.8029853Z  * [new branch]                gh/wconstab/468/orig    -> origin/gh/wconstab/468/orig
2025-12-04T12:26:06.8029926Z  * [new branch]                gh/weifengpy/39/base    -> origin/gh/weifengpy/39/base
2025-12-04T12:26:06.8029999Z  * [new branch]                gh/weifengpy/39/head    -> origin/gh/weifengpy/39/head
2025-12-04T12:26:06.8030071Z  * [new branch]                gh/weifengpy/39/orig    -> origin/gh/weifengpy/39/orig
2025-12-04T12:26:06.8030142Z  * [new branch]                gh/weifengpy/40/base    -> origin/gh/weifengpy/40/base
2025-12-04T12:26:06.8030214Z  * [new branch]                gh/weifengpy/40/head    -> origin/gh/weifengpy/40/head
2025-12-04T12:26:06.8030285Z  * [new branch]                gh/weifengpy/40/orig    -> origin/gh/weifengpy/40/orig
2025-12-04T12:26:06.8030356Z  * [new branch]                gh/weifengpy/41/base    -> origin/gh/weifengpy/41/base
2025-12-04T12:26:06.8030428Z  * [new branch]                gh/weifengpy/41/head    -> origin/gh/weifengpy/41/head
2025-12-04T12:26:06.8030522Z  * [new branch]                gh/weifengpy/41/orig    -> origin/gh/weifengpy/41/orig
2025-12-04T12:26:06.8030605Z  * [new branch]                gh/williamwen42/250/base -> origin/gh/williamwen42/250/base
2025-12-04T12:26:06.8030720Z  * [new branch]                gh/williamwen42/250/head -> origin/gh/williamwen42/250/head
2025-12-04T12:26:06.8030799Z  * [new branch]                gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig
2025-12-04T12:26:06.8030876Z  * [new branch]                gh/williamwen42/279/base -> origin/gh/williamwen42/279/base
2025-12-04T12:26:06.8030955Z  * [new branch]                gh/williamwen42/279/head -> origin/gh/williamwen42/279/head
2025-12-04T12:26:06.8031033Z  * [new branch]                gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig
2025-12-04T12:26:06.8031112Z  * [new branch]                gh/williamwen42/282/base -> origin/gh/williamwen42/282/base
2025-12-04T12:26:06.8031189Z  * [new branch]                gh/williamwen42/282/head -> origin/gh/williamwen42/282/head
2025-12-04T12:26:06.8031270Z  * [new branch]                gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig
2025-12-04T12:26:06.8031349Z  * [new branch]                gh/williamwen42/287/base -> origin/gh/williamwen42/287/base
2025-12-04T12:26:06.8031429Z  * [new branch]                gh/williamwen42/287/head -> origin/gh/williamwen42/287/head
2025-12-04T12:26:06.8031506Z  * [new branch]                gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig
2025-12-04T12:26:06.8031585Z  * [new branch]                gh/williamwen42/288/base -> origin/gh/williamwen42/288/base
2025-12-04T12:26:06.8031662Z  * [new branch]                gh/williamwen42/288/head -> origin/gh/williamwen42/288/head
2025-12-04T12:26:06.8031739Z  * [new branch]                gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig
2025-12-04T12:26:06.8031816Z  * [new branch]                gh/williamwen42/296/base -> origin/gh/williamwen42/296/base
2025-12-04T12:26:06.8031895Z  * [new branch]                gh/williamwen42/296/head -> origin/gh/williamwen42/296/head
2025-12-04T12:26:06.8031972Z  * [new branch]                gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig
2025-12-04T12:26:06.8032051Z  * [new branch]                gh/williamwen42/297/base -> origin/gh/williamwen42/297/base
2025-12-04T12:26:06.8032127Z  * [new branch]                gh/williamwen42/297/head -> origin/gh/williamwen42/297/head
2025-12-04T12:26:06.8032204Z  * [new branch]                gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig
2025-12-04T12:26:06.8032282Z  * [new branch]                gh/williamwen42/306/base -> origin/gh/williamwen42/306/base
2025-12-04T12:26:06.8032358Z  * [new branch]                gh/williamwen42/306/head -> origin/gh/williamwen42/306/head
2025-12-04T12:26:06.8032437Z  * [new branch]                gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig
2025-12-04T12:26:06.8032514Z  * [new branch]                gh/williamwen42/309/base -> origin/gh/williamwen42/309/base
2025-12-04T12:26:06.8032592Z  * [new branch]                gh/williamwen42/309/head -> origin/gh/williamwen42/309/head
2025-12-04T12:26:06.8032671Z  * [new branch]                gh/williamwen42/309/orig -> origin/gh/williamwen42/309/orig
2025-12-04T12:26:06.8032750Z  * [new branch]                gh/williamwen42/310/base -> origin/gh/williamwen42/310/base
2025-12-04T12:26:06.8032827Z  * [new branch]                gh/williamwen42/310/head -> origin/gh/williamwen42/310/head
2025-12-04T12:26:06.8032906Z  * [new branch]                gh/williamwen42/310/orig -> origin/gh/williamwen42/310/orig
2025-12-04T12:26:06.8032982Z  * [new branch]                gh/williamwen42/311/base -> origin/gh/williamwen42/311/base
2025-12-04T12:26:06.8033059Z  * [new branch]                gh/williamwen42/311/head -> origin/gh/williamwen42/311/head
2025-12-04T12:26:06.8033137Z  * [new branch]                gh/williamwen42/311/orig -> origin/gh/williamwen42/311/orig
2025-12-04T12:26:06.8033241Z  * [new branch]                gh/williamwen42/319/base -> origin/gh/williamwen42/319/base
2025-12-04T12:26:06.8033321Z  * [new branch]                gh/williamwen42/319/head -> origin/gh/williamwen42/319/head
2025-12-04T12:26:06.8033402Z  * [new branch]                gh/williamwen42/319/orig -> origin/gh/williamwen42/319/orig
2025-12-04T12:26:06.8033500Z  * [new branch]                gh/williamwen42/325/base -> origin/gh/williamwen42/325/base
2025-12-04T12:26:06.8033576Z  * [new branch]                gh/williamwen42/325/head -> origin/gh/williamwen42/325/head
2025-12-04T12:26:06.8033656Z  * [new branch]                gh/williamwen42/325/orig -> origin/gh/williamwen42/325/orig
2025-12-04T12:26:06.8033733Z  * [new branch]                gh/williamwen42/326/base -> origin/gh/williamwen42/326/base
2025-12-04T12:26:06.8033809Z  * [new branch]                gh/williamwen42/326/head -> origin/gh/williamwen42/326/head
2025-12-04T12:26:06.8033889Z  * [new branch]                gh/williamwen42/326/orig -> origin/gh/williamwen42/326/orig
2025-12-04T12:26:06.8033968Z  * [new branch]                gh/williamwen42/327/base -> origin/gh/williamwen42/327/base
2025-12-04T12:26:06.8034047Z  * [new branch]                gh/williamwen42/327/head -> origin/gh/williamwen42/327/head
2025-12-04T12:26:06.8034125Z  * [new branch]                gh/williamwen42/327/orig -> origin/gh/williamwen42/327/orig
2025-12-04T12:26:06.8034203Z  * [new branch]                gh/williamwen42/328/base -> origin/gh/williamwen42/328/base
2025-12-04T12:26:06.8034282Z  * [new branch]                gh/williamwen42/328/head -> origin/gh/williamwen42/328/head
2025-12-04T12:26:06.8034360Z  * [new branch]                gh/williamwen42/328/orig -> origin/gh/williamwen42/328/orig
2025-12-04T12:26:06.8034438Z  * [new branch]                gh/williamwen42/329/base -> origin/gh/williamwen42/329/base
2025-12-04T12:26:06.8034518Z  * [new branch]                gh/williamwen42/329/head -> origin/gh/williamwen42/329/head
2025-12-04T12:26:06.8034596Z  * [new branch]                gh/williamwen42/329/orig -> origin/gh/williamwen42/329/orig
2025-12-04T12:26:06.8034673Z  * [new branch]                gh/williamwen42/330/base -> origin/gh/williamwen42/330/base
2025-12-04T12:26:06.8034750Z  * [new branch]                gh/williamwen42/330/head -> origin/gh/williamwen42/330/head
2025-12-04T12:26:06.8034830Z  * [new branch]                gh/williamwen42/330/orig -> origin/gh/williamwen42/330/orig
2025-12-04T12:26:06.8034908Z  * [new branch]                gh/williamwen42/331/base -> origin/gh/williamwen42/331/base
2025-12-04T12:26:06.8034986Z  * [new branch]                gh/williamwen42/331/head -> origin/gh/williamwen42/331/head
2025-12-04T12:26:06.8035064Z  * [new branch]                gh/williamwen42/331/orig -> origin/gh/williamwen42/331/orig
2025-12-04T12:26:06.8035141Z  * [new branch]                gh/williamwen42/332/base -> origin/gh/williamwen42/332/base
2025-12-04T12:26:06.8035218Z  * [new branch]                gh/williamwen42/332/head -> origin/gh/williamwen42/332/head
2025-12-04T12:26:06.8035296Z  * [new branch]                gh/williamwen42/332/orig -> origin/gh/williamwen42/332/orig
2025-12-04T12:26:06.8035374Z  * [new branch]                gh/williamwen42/333/base -> origin/gh/williamwen42/333/base
2025-12-04T12:26:06.8035452Z  * [new branch]                gh/williamwen42/333/head -> origin/gh/williamwen42/333/head
2025-12-04T12:26:06.8035529Z  * [new branch]                gh/williamwen42/333/orig -> origin/gh/williamwen42/333/orig
2025-12-04T12:26:06.8035607Z  * [new branch]                gh/williamwen42/334/base -> origin/gh/williamwen42/334/base
2025-12-04T12:26:06.8035683Z  * [new branch]                gh/williamwen42/334/head -> origin/gh/williamwen42/334/head
2025-12-04T12:26:06.8035760Z  * [new branch]                gh/williamwen42/334/orig -> origin/gh/williamwen42/334/orig
2025-12-04T12:26:06.8035838Z  * [new branch]                gh/williamwen42/335/base -> origin/gh/williamwen42/335/base
2025-12-04T12:26:06.8035936Z  * [new branch]                gh/williamwen42/335/head -> origin/gh/williamwen42/335/head
2025-12-04T12:26:06.8036014Z  * [new branch]                gh/williamwen42/335/orig -> origin/gh/williamwen42/335/orig
2025-12-04T12:26:06.8036092Z  * [new branch]                gh/williamwen42/336/base -> origin/gh/williamwen42/336/base
2025-12-04T12:26:06.8036189Z  * [new branch]                gh/williamwen42/336/head -> origin/gh/williamwen42/336/head
2025-12-04T12:26:06.8036265Z  * [new branch]                gh/williamwen42/336/orig -> origin/gh/williamwen42/336/orig
2025-12-04T12:26:06.8036344Z  * [new branch]                gh/williamwen42/337/base -> origin/gh/williamwen42/337/base
2025-12-04T12:26:06.8036420Z  * [new branch]                gh/williamwen42/337/head -> origin/gh/williamwen42/337/head
2025-12-04T12:26:06.8036497Z  * [new branch]                gh/williamwen42/337/orig -> origin/gh/williamwen42/337/orig
2025-12-04T12:26:06.8036576Z  * [new branch]                gh/williamwen42/338/base -> origin/gh/williamwen42/338/base
2025-12-04T12:26:06.8036655Z  * [new branch]                gh/williamwen42/338/head -> origin/gh/williamwen42/338/head
2025-12-04T12:26:06.8036732Z  * [new branch]                gh/williamwen42/338/orig -> origin/gh/williamwen42/338/orig
2025-12-04T12:26:06.8036846Z  * [new branch]                gh/williamwen42/339/base -> origin/gh/williamwen42/339/base
2025-12-04T12:26:06.8036923Z  * [new branch]                gh/williamwen42/339/head -> origin/gh/williamwen42/339/head
2025-12-04T12:26:06.8037001Z  * [new branch]                gh/williamwen42/339/orig -> origin/gh/williamwen42/339/orig
2025-12-04T12:26:06.8037077Z  * [new branch]                gh/williamwen42/340/base -> origin/gh/williamwen42/340/base
2025-12-04T12:26:06.8037153Z  * [new branch]                gh/williamwen42/340/head -> origin/gh/williamwen42/340/head
2025-12-04T12:26:06.8037230Z  * [new branch]                gh/williamwen42/340/orig -> origin/gh/williamwen42/340/orig
2025-12-04T12:26:06.8037308Z  * [new branch]                gh/williamwen42/341/base -> origin/gh/williamwen42/341/base
2025-12-04T12:26:06.8037386Z  * [new branch]                gh/williamwen42/341/head -> origin/gh/williamwen42/341/head
2025-12-04T12:26:06.8037464Z  * [new branch]                gh/williamwen42/341/orig -> origin/gh/williamwen42/341/orig
2025-12-04T12:26:06.8037542Z  * [new branch]                gh/williamwen42/342/base -> origin/gh/williamwen42/342/base
2025-12-04T12:26:06.8037618Z  * [new branch]                gh/williamwen42/342/head -> origin/gh/williamwen42/342/head
2025-12-04T12:26:06.8037697Z  * [new branch]                gh/williamwen42/342/orig -> origin/gh/williamwen42/342/orig
2025-12-04T12:26:06.8037774Z  * [new branch]                gh/williamwen42/343/base -> origin/gh/williamwen42/343/base
2025-12-04T12:26:06.8037851Z  * [new branch]                gh/williamwen42/343/head -> origin/gh/williamwen42/343/head
2025-12-04T12:26:06.8037929Z  * [new branch]                gh/williamwen42/343/orig -> origin/gh/williamwen42/343/orig
2025-12-04T12:26:06.8038007Z  * [new branch]                gh/williamwen42/344/base -> origin/gh/williamwen42/344/base
2025-12-04T12:26:06.8038084Z  * [new branch]                gh/williamwen42/344/head -> origin/gh/williamwen42/344/head
2025-12-04T12:26:06.8038163Z  * [new branch]                gh/williamwen42/344/orig -> origin/gh/williamwen42/344/orig
2025-12-04T12:26:06.8038240Z  * [new branch]                gh/williamwen42/345/base -> origin/gh/williamwen42/345/base
2025-12-04T12:26:06.8038318Z  * [new branch]                gh/williamwen42/345/head -> origin/gh/williamwen42/345/head
2025-12-04T12:26:06.8038395Z  * [new branch]                gh/williamwen42/345/orig -> origin/gh/williamwen42/345/orig
2025-12-04T12:26:06.8038471Z  * [new branch]                gh/williamwen42/346/base -> origin/gh/williamwen42/346/base
2025-12-04T12:26:06.8038549Z  * [new branch]                gh/williamwen42/346/head -> origin/gh/williamwen42/346/head
2025-12-04T12:26:06.8038665Z  * [new branch]                gh/williamwen42/346/orig -> origin/gh/williamwen42/346/orig
2025-12-04T12:26:06.8038743Z  * [new branch]                gh/williamwen42/347/base -> origin/gh/williamwen42/347/base
2025-12-04T12:26:06.8038821Z  * [new branch]                gh/williamwen42/347/head -> origin/gh/williamwen42/347/head
2025-12-04T12:26:06.8038930Z  * [new branch]                gh/williamwen42/347/orig -> origin/gh/williamwen42/347/orig
2025-12-04T12:26:06.8039007Z  * [new branch]                gh/williamwen42/348/base -> origin/gh/williamwen42/348/base
2025-12-04T12:26:06.8039086Z  * [new branch]                gh/williamwen42/348/head -> origin/gh/williamwen42/348/head
2025-12-04T12:26:06.8039162Z  * [new branch]                gh/williamwen42/348/orig -> origin/gh/williamwen42/348/orig
2025-12-04T12:26:06.8039239Z  * [new branch]                gh/williamwen42/349/base -> origin/gh/williamwen42/349/base
2025-12-04T12:26:06.8039316Z  * [new branch]                gh/williamwen42/349/head -> origin/gh/williamwen42/349/head
2025-12-04T12:26:06.8039394Z  * [new branch]                gh/williamwen42/349/orig -> origin/gh/williamwen42/349/orig
2025-12-04T12:26:06.8039471Z  * [new branch]                gh/williamwen42/350/base -> origin/gh/williamwen42/350/base
2025-12-04T12:26:06.8039551Z  * [new branch]                gh/williamwen42/350/head -> origin/gh/williamwen42/350/head
2025-12-04T12:26:06.8039629Z  * [new branch]                gh/williamwen42/350/orig -> origin/gh/williamwen42/350/orig
2025-12-04T12:26:06.8039705Z  * [new branch]                gh/williamwen42/351/base -> origin/gh/williamwen42/351/base
2025-12-04T12:26:06.8039782Z  * [new branch]                gh/williamwen42/351/head -> origin/gh/williamwen42/351/head
2025-12-04T12:26:06.8039859Z  * [new branch]                gh/williamwen42/351/orig -> origin/gh/williamwen42/351/orig
2025-12-04T12:26:06.8039936Z  * [new branch]                gh/williamwen42/352/base -> origin/gh/williamwen42/352/base
2025-12-04T12:26:06.8040014Z  * [new branch]                gh/williamwen42/352/head -> origin/gh/williamwen42/352/head
2025-12-04T12:26:06.8040091Z  * [new branch]                gh/williamwen42/352/orig -> origin/gh/williamwen42/352/orig
2025-12-04T12:26:06.8040169Z  * [new branch]                gh/williamwen42/353/base -> origin/gh/williamwen42/353/base
2025-12-04T12:26:06.8040247Z  * [new branch]                gh/williamwen42/353/head -> origin/gh/williamwen42/353/head
2025-12-04T12:26:06.8040324Z  * [new branch]                gh/williamwen42/353/orig -> origin/gh/williamwen42/353/orig
2025-12-04T12:26:06.8040402Z  * [new branch]                gh/williamwen42/354/base -> origin/gh/williamwen42/354/base
2025-12-04T12:26:06.8040478Z  * [new branch]                gh/williamwen42/354/head -> origin/gh/williamwen42/354/head
2025-12-04T12:26:06.8040555Z  * [new branch]                gh/williamwen42/354/orig -> origin/gh/williamwen42/354/orig
2025-12-04T12:26:06.8040633Z  * [new branch]                gh/williamwen42/355/base -> origin/gh/williamwen42/355/base
2025-12-04T12:26:06.8040711Z  * [new branch]                gh/williamwen42/355/head -> origin/gh/williamwen42/355/head
2025-12-04T12:26:06.8040788Z  * [new branch]                gh/williamwen42/355/orig -> origin/gh/williamwen42/355/orig
2025-12-04T12:26:06.8040867Z  * [new branch]                gh/williamwen42/356/base -> origin/gh/williamwen42/356/base
2025-12-04T12:26:06.8040944Z  * [new branch]                gh/williamwen42/356/head -> origin/gh/williamwen42/356/head
2025-12-04T12:26:06.8041021Z  * [new branch]                gh/williamwen42/356/orig -> origin/gh/williamwen42/356/orig
2025-12-04T12:26:06.8041098Z  * [new branch]                gh/williamwen42/357/base -> origin/gh/williamwen42/357/base
2025-12-04T12:26:06.8041174Z  * [new branch]                gh/williamwen42/357/head -> origin/gh/williamwen42/357/head
2025-12-04T12:26:06.8041252Z  * [new branch]                gh/williamwen42/357/orig -> origin/gh/williamwen42/357/orig
2025-12-04T12:26:06.8041328Z  * [new branch]                gh/williamwen42/358/base -> origin/gh/williamwen42/358/base
2025-12-04T12:26:06.8041433Z  * [new branch]                gh/williamwen42/358/head -> origin/gh/williamwen42/358/head
2025-12-04T12:26:06.8041514Z  * [new branch]                gh/williamwen42/358/orig -> origin/gh/williamwen42/358/orig
2025-12-04T12:26:06.8041621Z  * [new branch]                gh/xmfan/169/base       -> origin/gh/xmfan/169/base
2025-12-04T12:26:06.8041689Z  * [new branch]                gh/xmfan/169/head       -> origin/gh/xmfan/169/head
2025-12-04T12:26:06.8041758Z  * [new branch]                gh/xmfan/170/base       -> origin/gh/xmfan/170/base
2025-12-04T12:26:06.8041825Z  * [new branch]                gh/xmfan/170/head       -> origin/gh/xmfan/170/head
2025-12-04T12:26:06.8041890Z  * [new branch]                gh/xmfan/274/base       -> origin/gh/xmfan/274/base
2025-12-04T12:26:06.8041958Z  * [new branch]                gh/xmfan/274/head       -> origin/gh/xmfan/274/head
2025-12-04T12:26:06.8042023Z  * [new branch]                gh/xmfan/274/orig       -> origin/gh/xmfan/274/orig
2025-12-04T12:26:06.8042089Z  * [new branch]                gh/xmfan/277/base       -> origin/gh/xmfan/277/base
2025-12-04T12:26:06.8042156Z  * [new branch]                gh/xmfan/277/head       -> origin/gh/xmfan/277/head
2025-12-04T12:26:06.8042224Z  * [new branch]                gh/xmfan/277/orig       -> origin/gh/xmfan/277/orig
2025-12-04T12:26:06.8042290Z  * [new branch]                gh/xmfan/301/base       -> origin/gh/xmfan/301/base
2025-12-04T12:26:06.8042356Z  * [new branch]                gh/xmfan/301/head       -> origin/gh/xmfan/301/head
2025-12-04T12:26:06.8042421Z  * [new branch]                gh/xmfan/301/orig       -> origin/gh/xmfan/301/orig
2025-12-04T12:26:06.8042485Z  * [new branch]                gh/xmfan/304/base       -> origin/gh/xmfan/304/base
2025-12-04T12:26:06.8042552Z  * [new branch]                gh/xmfan/304/head       -> origin/gh/xmfan/304/head
2025-12-04T12:26:06.8042617Z  * [new branch]                gh/xmfan/304/orig       -> origin/gh/xmfan/304/orig
2025-12-04T12:26:06.8042685Z  * [new branch]                gh/xmfan/309/base       -> origin/gh/xmfan/309/base
2025-12-04T12:26:06.8042753Z  * [new branch]                gh/xmfan/309/head       -> origin/gh/xmfan/309/head
2025-12-04T12:26:06.8042820Z  * [new branch]                gh/xmfan/309/orig       -> origin/gh/xmfan/309/orig
2025-12-04T12:26:06.8042886Z  * [new branch]                gh/xmfan/310/base       -> origin/gh/xmfan/310/base
2025-12-04T12:26:06.8042951Z  * [new branch]                gh/xmfan/310/head       -> origin/gh/xmfan/310/head
2025-12-04T12:26:06.8043016Z  * [new branch]                gh/xmfan/310/orig       -> origin/gh/xmfan/310/orig
2025-12-04T12:26:06.8043083Z  * [new branch]                gh/xmfan/311/base       -> origin/gh/xmfan/311/base
2025-12-04T12:26:06.8043149Z  * [new branch]                gh/xmfan/311/head       -> origin/gh/xmfan/311/head
2025-12-04T12:26:06.8043214Z  * [new branch]                gh/xmfan/311/orig       -> origin/gh/xmfan/311/orig
2025-12-04T12:26:06.8043281Z  * [new branch]                gh/xmfan/312/base       -> origin/gh/xmfan/312/base
2025-12-04T12:26:06.8043347Z  * [new branch]                gh/xmfan/312/head       -> origin/gh/xmfan/312/head
2025-12-04T12:26:06.8043414Z  * [new branch]                gh/xmfan/312/orig       -> origin/gh/xmfan/312/orig
2025-12-04T12:26:06.8043481Z  * [new branch]                gh/xmfan/313/base       -> origin/gh/xmfan/313/base
2025-12-04T12:26:06.8043546Z  * [new branch]                gh/xmfan/313/head       -> origin/gh/xmfan/313/head
2025-12-04T12:26:06.8043611Z  * [new branch]                gh/xmfan/313/orig       -> origin/gh/xmfan/313/orig
2025-12-04T12:26:06.8043689Z  * [new branch]                gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base
2025-12-04T12:26:06.8043767Z  * [new branch]                gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head
2025-12-04T12:26:06.8043842Z  * [new branch]                gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig
2025-12-04T12:26:06.8043948Z  * [new branch]                gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base
2025-12-04T12:26:06.8044023Z  * [new branch]                gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head
2025-12-04T12:26:06.8044118Z  * [new branch]                gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig
2025-12-04T12:26:06.8044194Z  * [new branch]                gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base
2025-12-04T12:26:06.8044268Z  * [new branch]                gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head
2025-12-04T12:26:06.8044342Z  * [new branch]                gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig
2025-12-04T12:26:06.8044417Z  * [new branch]                gh/xuanzhang816/34/base -> origin/gh/xuanzhang816/34/base
2025-12-04T12:26:06.8044491Z  * [new branch]                gh/xuanzhang816/34/head -> origin/gh/xuanzhang816/34/head
2025-12-04T12:26:06.8044567Z  * [new branch]                gh/xuanzhang816/34/orig -> origin/gh/xuanzhang816/34/orig
2025-12-04T12:26:06.8044644Z  * [new branch]                gh/xuanzhang816/35/base -> origin/gh/xuanzhang816/35/base
2025-12-04T12:26:06.8044717Z  * [new branch]                gh/xuanzhang816/35/head -> origin/gh/xuanzhang816/35/head
2025-12-04T12:26:06.8044794Z  * [new branch]                gh/xuanzhang816/35/orig -> origin/gh/xuanzhang816/35/orig
2025-12-04T12:26:06.8044867Z  * [new branch]                gh/yanbing-j/11/base    -> origin/gh/yanbing-j/11/base
2025-12-04T12:26:06.8044939Z  * [new branch]                gh/yanbing-j/11/head    -> origin/gh/yanbing-j/11/head
2025-12-04T12:26:06.8045010Z  * [new branch]                gh/yanbing-j/11/orig    -> origin/gh/yanbing-j/11/orig
2025-12-04T12:26:06.8045081Z  * [new branch]                gh/yanbing-j/12/base    -> origin/gh/yanbing-j/12/base
2025-12-04T12:26:06.8045150Z  * [new branch]                gh/yanbing-j/12/head    -> origin/gh/yanbing-j/12/head
2025-12-04T12:26:06.8045221Z  * [new branch]                gh/yanbing-j/12/orig    -> origin/gh/yanbing-j/12/orig
2025-12-04T12:26:06.8045290Z  * [new branch]                gh/yanbing-j/13/base    -> origin/gh/yanbing-j/13/base
2025-12-04T12:26:06.8045360Z  * [new branch]                gh/yanbing-j/13/head    -> origin/gh/yanbing-j/13/head
2025-12-04T12:26:06.8045431Z  * [new branch]                gh/yanbing-j/13/orig    -> origin/gh/yanbing-j/13/orig
2025-12-04T12:26:06.8045500Z  * [new branch]                gh/yanbing-j/14/base    -> origin/gh/yanbing-j/14/base
2025-12-04T12:26:06.8045569Z  * [new branch]                gh/yanbing-j/14/head    -> origin/gh/yanbing-j/14/head
2025-12-04T12:26:06.8045640Z  * [new branch]                gh/yanbing-j/14/orig    -> origin/gh/yanbing-j/14/orig
2025-12-04T12:26:06.8045708Z  * [new branch]                gh/yanbing-j/15/base    -> origin/gh/yanbing-j/15/base
2025-12-04T12:26:06.8045776Z  * [new branch]                gh/yanbing-j/15/head    -> origin/gh/yanbing-j/15/head
2025-12-04T12:26:06.8045848Z  * [new branch]                gh/yanbing-j/15/orig    -> origin/gh/yanbing-j/15/orig
2025-12-04T12:26:06.8045917Z  * [new branch]                gh/yanbing-j/18/base    -> origin/gh/yanbing-j/18/base
2025-12-04T12:26:06.8045985Z  * [new branch]                gh/yanbing-j/18/head    -> origin/gh/yanbing-j/18/head
2025-12-04T12:26:06.8046057Z  * [new branch]                gh/yanbing-j/18/orig    -> origin/gh/yanbing-j/18/orig
2025-12-04T12:26:06.8046125Z  * [new branch]                gh/yanbing-j/19/base    -> origin/gh/yanbing-j/19/base
2025-12-04T12:26:06.8046196Z  * [new branch]                gh/yanbing-j/19/head    -> origin/gh/yanbing-j/19/head
2025-12-04T12:26:06.8046265Z  * [new branch]                gh/yanbing-j/19/orig    -> origin/gh/yanbing-j/19/orig
2025-12-04T12:26:06.8046333Z  * [new branch]                gh/yanbing-j/20/base    -> origin/gh/yanbing-j/20/base
2025-12-04T12:26:06.8046403Z  * [new branch]                gh/yanbing-j/20/head    -> origin/gh/yanbing-j/20/head
2025-12-04T12:26:06.8046495Z  * [new branch]                gh/yanbing-j/20/orig    -> origin/gh/yanbing-j/20/orig
2025-12-04T12:26:06.8046566Z  * [new branch]                gh/yanbing-j/21/base    -> origin/gh/yanbing-j/21/base
2025-12-04T12:26:06.8046640Z  * [new branch]                gh/yanbing-j/21/head    -> origin/gh/yanbing-j/21/head
2025-12-04T12:26:06.8046792Z  * [new branch]                gh/yanbing-j/22/base    -> origin/gh/yanbing-j/22/base
2025-12-04T12:26:06.8046865Z  * [new branch]                gh/yanbing-j/22/head    -> origin/gh/yanbing-j/22/head
2025-12-04T12:26:06.8046935Z  * [new branch]                gh/yanbing-j/22/orig    -> origin/gh/yanbing-j/22/orig
2025-12-04T12:26:06.8047005Z  * [new branch]                gh/yanbing-j/23/base    -> origin/gh/yanbing-j/23/base
2025-12-04T12:26:06.8047075Z  * [new branch]                gh/yanbing-j/23/head    -> origin/gh/yanbing-j/23/head
2025-12-04T12:26:06.8047146Z  * [new branch]                gh/yanbing-j/23/orig    -> origin/gh/yanbing-j/23/orig
2025-12-04T12:26:06.8047216Z  * [new branch]                gh/yanbing-j/24/base    -> origin/gh/yanbing-j/24/base
2025-12-04T12:26:06.8047285Z  * [new branch]                gh/yanbing-j/24/head    -> origin/gh/yanbing-j/24/head
2025-12-04T12:26:06.8047359Z  * [new branch]                gh/yanbing-j/24/orig    -> origin/gh/yanbing-j/24/orig
2025-12-04T12:26:06.8047427Z  * [new branch]                gh/yanbing-j/25/base    -> origin/gh/yanbing-j/25/base
2025-12-04T12:26:06.8047497Z  * [new branch]                gh/yanbing-j/25/head    -> origin/gh/yanbing-j/25/head
2025-12-04T12:26:06.8047568Z  * [new branch]                gh/yanbing-j/25/orig    -> origin/gh/yanbing-j/25/orig
2025-12-04T12:26:06.8047636Z  * [new branch]                gh/yanbing-j/26/base    -> origin/gh/yanbing-j/26/base
2025-12-04T12:26:06.8047706Z  * [new branch]                gh/yanbing-j/26/head    -> origin/gh/yanbing-j/26/head
2025-12-04T12:26:06.8047776Z  * [new branch]                gh/yanbing-j/26/orig    -> origin/gh/yanbing-j/26/orig
2025-12-04T12:26:06.8047857Z  * [new branch]                gh/yang-yu-hang/1/base  -> origin/gh/yang-yu-hang/1/base
2025-12-04T12:26:06.8047936Z  * [new branch]                gh/yang-yu-hang/1/head  -> origin/gh/yang-yu-hang/1/head
2025-12-04T12:26:06.8048012Z  * [new branch]                gh/yang-yu-hang/1/orig  -> origin/gh/yang-yu-hang/1/orig
2025-12-04T12:26:06.8048084Z  * [new branch]                gh/yang-yu-hang/2/base  -> origin/gh/yang-yu-hang/2/base
2025-12-04T12:26:06.8048157Z  * [new branch]                gh/yang-yu-hang/2/head  -> origin/gh/yang-yu-hang/2/head
2025-12-04T12:26:06.8048229Z  * [new branch]                gh/yang-yu-hang/2/orig  -> origin/gh/yang-yu-hang/2/orig
2025-12-04T12:26:06.8048302Z  * [new branch]                gh/yang-yu-hang/3/base  -> origin/gh/yang-yu-hang/3/base
2025-12-04T12:26:06.8048375Z  * [new branch]                gh/yang-yu-hang/3/head  -> origin/gh/yang-yu-hang/3/head
2025-12-04T12:26:06.8048447Z  * [new branch]                gh/yang-yu-hang/3/orig  -> origin/gh/yang-yu-hang/3/orig
2025-12-04T12:26:06.8048522Z  * [new branch]                gh/yangw-dev/12/base    -> origin/gh/yangw-dev/12/base
2025-12-04T12:26:06.8048595Z  * [new branch]                gh/yangw-dev/12/head    -> origin/gh/yangw-dev/12/head
2025-12-04T12:26:06.8048667Z  * [new branch]                gh/yangw-dev/12/orig    -> origin/gh/yangw-dev/12/orig
2025-12-04T12:26:06.8048737Z  * [new branch]                gh/yangw-dev/13/base    -> origin/gh/yangw-dev/13/base
2025-12-04T12:26:06.8048809Z  * [new branch]                gh/yangw-dev/13/head    -> origin/gh/yangw-dev/13/head
2025-12-04T12:26:06.8048878Z  * [new branch]                gh/yangw-dev/13/orig    -> origin/gh/yangw-dev/13/orig
2025-12-04T12:26:06.8048947Z  * [new branch]                gh/yangw-dev/14/base    -> origin/gh/yangw-dev/14/base
2025-12-04T12:26:06.8049018Z  * [new branch]                gh/yangw-dev/14/head    -> origin/gh/yangw-dev/14/head
2025-12-04T12:26:06.8049088Z  * [new branch]                gh/yangw-dev/14/orig    -> origin/gh/yangw-dev/14/orig
2025-12-04T12:26:06.8049204Z  * [new branch]                gh/yangw-dev/15/base    -> origin/gh/yangw-dev/15/base
2025-12-04T12:26:06.8049276Z  * [new branch]                gh/yangw-dev/15/head    -> origin/gh/yangw-dev/15/head
2025-12-04T12:26:06.8049381Z  * [new branch]                gh/yangw-dev/15/orig    -> origin/gh/yangw-dev/15/orig
2025-12-04T12:26:06.8049455Z  * [new branch]                gh/yangw-dev/19/base    -> origin/gh/yangw-dev/19/base
2025-12-04T12:26:06.8073327Z  * [new branch]                gh/yangw-dev/19/head    -> origin/gh/yangw-dev/19/head
2025-12-04T12:26:06.8073441Z  * [new branch]                gh/yangw-dev/19/orig    -> origin/gh/yangw-dev/19/orig
2025-12-04T12:26:06.8073521Z  * [new branch]                gh/yangw-dev/26/base    -> origin/gh/yangw-dev/26/base
2025-12-04T12:26:06.8073597Z  * [new branch]                gh/yangw-dev/26/head    -> origin/gh/yangw-dev/26/head
2025-12-04T12:26:06.8073683Z  * [new branch]                gh/yangw-dev/26/orig    -> origin/gh/yangw-dev/26/orig
2025-12-04T12:26:06.8073756Z  * [new branch]                gh/yangw-dev/27/base    -> origin/gh/yangw-dev/27/base
2025-12-04T12:26:06.8073827Z  * [new branch]                gh/yangw-dev/27/head    -> origin/gh/yangw-dev/27/head
2025-12-04T12:26:06.8073900Z  * [new branch]                gh/yangw-dev/27/orig    -> origin/gh/yangw-dev/27/orig
2025-12-04T12:26:06.8073975Z  * [new branch]                gh/ydwu4/292/base       -> origin/gh/ydwu4/292/base
2025-12-04T12:26:06.8074044Z  * [new branch]                gh/ydwu4/292/head       -> origin/gh/ydwu4/292/head
2025-12-04T12:26:06.8074111Z  * [new branch]                gh/ydwu4/292/orig       -> origin/gh/ydwu4/292/orig
2025-12-04T12:26:06.8074178Z  * [new branch]                gh/ydwu4/294/base       -> origin/gh/ydwu4/294/base
2025-12-04T12:26:06.8074243Z  * [new branch]                gh/ydwu4/294/head       -> origin/gh/ydwu4/294/head
2025-12-04T12:26:06.8074309Z  * [new branch]                gh/ydwu4/294/orig       -> origin/gh/ydwu4/294/orig
2025-12-04T12:26:06.8074378Z  * [new branch]                gh/ydwu4/295/base       -> origin/gh/ydwu4/295/base
2025-12-04T12:26:06.8074444Z  * [new branch]                gh/ydwu4/295/head       -> origin/gh/ydwu4/295/head
2025-12-04T12:26:06.8074511Z  * [new branch]                gh/ydwu4/295/orig       -> origin/gh/ydwu4/295/orig
2025-12-04T12:26:06.8074578Z  * [new branch]                gh/ydwu4/296/base       -> origin/gh/ydwu4/296/base
2025-12-04T12:26:06.8074644Z  * [new branch]                gh/ydwu4/296/head       -> origin/gh/ydwu4/296/head
2025-12-04T12:26:06.8074710Z  * [new branch]                gh/ydwu4/296/orig       -> origin/gh/ydwu4/296/orig
2025-12-04T12:26:06.8074775Z  * [new branch]                gh/ydwu4/306/base       -> origin/gh/ydwu4/306/base
2025-12-04T12:26:06.8074840Z  * [new branch]                gh/ydwu4/306/head       -> origin/gh/ydwu4/306/head
2025-12-04T12:26:06.8074906Z  * [new branch]                gh/ydwu4/306/orig       -> origin/gh/ydwu4/306/orig
2025-12-04T12:26:06.8074973Z  * [new branch]                gh/ydwu4/312/base       -> origin/gh/ydwu4/312/base
2025-12-04T12:26:06.8075038Z  * [new branch]                gh/ydwu4/312/head       -> origin/gh/ydwu4/312/head
2025-12-04T12:26:06.8075106Z  * [new branch]                gh/ydwu4/312/orig       -> origin/gh/ydwu4/312/orig
2025-12-04T12:26:06.8075172Z  * [new branch]                gh/ydwu4/322/base       -> origin/gh/ydwu4/322/base
2025-12-04T12:26:06.8075237Z  * [new branch]                gh/ydwu4/322/head       -> origin/gh/ydwu4/322/head
2025-12-04T12:26:06.8075303Z  * [new branch]                gh/ydwu4/322/orig       -> origin/gh/ydwu4/322/orig
2025-12-04T12:26:06.8075371Z  * [new branch]                gh/ydwu4/327/base       -> origin/gh/ydwu4/327/base
2025-12-04T12:26:06.8075436Z  * [new branch]                gh/ydwu4/327/head       -> origin/gh/ydwu4/327/head
2025-12-04T12:26:06.8075502Z  * [new branch]                gh/ydwu4/327/orig       -> origin/gh/ydwu4/327/orig
2025-12-04T12:26:06.8075650Z  * [new branch]                gh/ydwu4/328/base       -> origin/gh/ydwu4/328/base
2025-12-04T12:26:06.8075717Z  * [new branch]                gh/ydwu4/328/head       -> origin/gh/ydwu4/328/head
2025-12-04T12:26:06.8075822Z  * [new branch]                gh/ydwu4/328/orig       -> origin/gh/ydwu4/328/orig
2025-12-04T12:26:06.8075888Z  * [new branch]                gh/ydwu4/329/base       -> origin/gh/ydwu4/329/base
2025-12-04T12:26:06.8075953Z  * [new branch]                gh/ydwu4/329/head       -> origin/gh/ydwu4/329/head
2025-12-04T12:26:06.8076020Z  * [new branch]                gh/ydwu4/329/orig       -> origin/gh/ydwu4/329/orig
2025-12-04T12:26:06.8076085Z  * [new branch]                gh/ydwu4/330/base       -> origin/gh/ydwu4/330/base
2025-12-04T12:26:06.8076150Z  * [new branch]                gh/ydwu4/330/head       -> origin/gh/ydwu4/330/head
2025-12-04T12:26:06.8076216Z  * [new branch]                gh/ydwu4/330/orig       -> origin/gh/ydwu4/330/orig
2025-12-04T12:26:06.8076282Z  * [new branch]                gh/ydwu4/331/base       -> origin/gh/ydwu4/331/base
2025-12-04T12:26:06.8076349Z  * [new branch]                gh/ydwu4/331/head       -> origin/gh/ydwu4/331/head
2025-12-04T12:26:06.8076418Z  * [new branch]                gh/ydwu4/331/orig       -> origin/gh/ydwu4/331/orig
2025-12-04T12:26:06.8076484Z  * [new branch]                gh/ydwu4/332/base       -> origin/gh/ydwu4/332/base
2025-12-04T12:26:06.8076550Z  * [new branch]                gh/ydwu4/332/head       -> origin/gh/ydwu4/332/head
2025-12-04T12:26:06.8076616Z  * [new branch]                gh/ydwu4/332/orig       -> origin/gh/ydwu4/332/orig
2025-12-04T12:26:06.8076681Z  * [new branch]                gh/ydwu4/333/base       -> origin/gh/ydwu4/333/base
2025-12-04T12:26:06.8076793Z  * [new branch]                gh/ydwu4/333/head       -> origin/gh/ydwu4/333/head
2025-12-04T12:26:06.8076864Z  * [new branch]                gh/ydwu4/333/orig       -> origin/gh/ydwu4/333/orig
2025-12-04T12:26:06.8076931Z  * [new branch]                gh/ydwu4/334/base       -> origin/gh/ydwu4/334/base
2025-12-04T12:26:06.8076998Z  * [new branch]                gh/ydwu4/334/head       -> origin/gh/ydwu4/334/head
2025-12-04T12:26:06.8077064Z  * [new branch]                gh/ydwu4/334/orig       -> origin/gh/ydwu4/334/orig
2025-12-04T12:26:06.8077132Z  * [new branch]                gh/ydwu4/335/base       -> origin/gh/ydwu4/335/base
2025-12-04T12:26:06.8077198Z  * [new branch]                gh/ydwu4/335/head       -> origin/gh/ydwu4/335/head
2025-12-04T12:26:06.8077264Z  * [new branch]                gh/ydwu4/335/orig       -> origin/gh/ydwu4/335/orig
2025-12-04T12:26:06.8077330Z  * [new branch]                gh/ydwu4/337/base       -> origin/gh/ydwu4/337/base
2025-12-04T12:26:06.8077400Z  * [new branch]                gh/ydwu4/337/head       -> origin/gh/ydwu4/337/head
2025-12-04T12:26:06.8077466Z  * [new branch]                gh/ydwu4/337/orig       -> origin/gh/ydwu4/337/orig
2025-12-04T12:26:06.8077533Z  * [new branch]                gh/ydwu4/339/base       -> origin/gh/ydwu4/339/base
2025-12-04T12:26:06.8077602Z  * [new branch]                gh/ydwu4/339/head       -> origin/gh/ydwu4/339/head
2025-12-04T12:26:06.8077669Z  * [new branch]                gh/ydwu4/339/orig       -> origin/gh/ydwu4/339/orig
2025-12-04T12:26:06.8077735Z  * [new branch]                gh/yf225/133/base       -> origin/gh/yf225/133/base
2025-12-04T12:26:06.8077801Z  * [new branch]                gh/yf225/133/head       -> origin/gh/yf225/133/head
2025-12-04T12:26:06.8077868Z  * [new branch]                gh/yf225/93/base        -> origin/gh/yf225/93/base
2025-12-04T12:26:06.8077933Z  * [new branch]                gh/yf225/93/head        -> origin/gh/yf225/93/head
2025-12-04T12:26:06.8078010Z  * [new branch]                gh/yifuwang/152/base    -> origin/gh/yifuwang/152/base
2025-12-04T12:26:06.8078084Z  * [new branch]                gh/yifuwang/152/head    -> origin/gh/yifuwang/152/head
2025-12-04T12:26:06.8078203Z  * [new branch]                gh/yifuwang/152/orig    -> origin/gh/yifuwang/152/orig
2025-12-04T12:26:06.8078277Z  * [new branch]                gh/yifuwang/195/base    -> origin/gh/yifuwang/195/base
2025-12-04T12:26:06.8078348Z  * [new branch]                gh/yifuwang/195/head    -> origin/gh/yifuwang/195/head
2025-12-04T12:26:06.8078455Z  * [new branch]                gh/yifuwang/195/orig    -> origin/gh/yifuwang/195/orig
2025-12-04T12:26:06.8078527Z  * [new branch]                gh/yiming0416/1/base    -> origin/gh/yiming0416/1/base
2025-12-04T12:26:06.8078599Z  * [new branch]                gh/yiming0416/1/head    -> origin/gh/yiming0416/1/head
2025-12-04T12:26:06.8078669Z  * [new branch]                gh/yiming0416/2/base    -> origin/gh/yiming0416/2/base
2025-12-04T12:26:06.8078741Z  * [new branch]                gh/yiming0416/2/head    -> origin/gh/yiming0416/2/head
2025-12-04T12:26:06.8078818Z  * [new branch]                gh/yushangdi/1/base     -> origin/gh/yushangdi/1/base
2025-12-04T12:26:06.8078896Z  * [new branch]                gh/yushangdi/1/head     -> origin/gh/yushangdi/1/head
2025-12-04T12:26:06.8078968Z  * [new branch]                gh/yushangdi/10/base    -> origin/gh/yushangdi/10/base
2025-12-04T12:26:06.8079042Z  * [new branch]                gh/yushangdi/10/head    -> origin/gh/yushangdi/10/head
2025-12-04T12:26:06.8079120Z  * [new branch]                gh/yushangdi/10/orig    -> origin/gh/yushangdi/10/orig
2025-12-04T12:26:06.8079190Z  * [new branch]                gh/yushangdi/11/base    -> origin/gh/yushangdi/11/base
2025-12-04T12:26:06.8079260Z  * [new branch]                gh/yushangdi/11/head    -> origin/gh/yushangdi/11/head
2025-12-04T12:26:06.8079332Z  * [new branch]                gh/yushangdi/11/orig    -> origin/gh/yushangdi/11/orig
2025-12-04T12:26:06.8079402Z  * [new branch]                gh/yushangdi/2/base     -> origin/gh/yushangdi/2/base
2025-12-04T12:26:06.8079474Z  * [new branch]                gh/yushangdi/2/head     -> origin/gh/yushangdi/2/head
2025-12-04T12:26:06.8079548Z  * [new branch]                gh/yushangdi/7/base     -> origin/gh/yushangdi/7/base
2025-12-04T12:26:06.8079618Z  * [new branch]                gh/yushangdi/7/head     -> origin/gh/yushangdi/7/head
2025-12-04T12:26:06.8079689Z  * [new branch]                gh/yushangdi/7/orig     -> origin/gh/yushangdi/7/orig
2025-12-04T12:26:06.8079761Z  * [new branch]                gh/yushangdi/8/base     -> origin/gh/yushangdi/8/base
2025-12-04T12:26:06.8079832Z  * [new branch]                gh/yushangdi/8/head     -> origin/gh/yushangdi/8/head
2025-12-04T12:26:06.8079902Z  * [new branch]                gh/yushangdi/8/orig     -> origin/gh/yushangdi/8/orig
2025-12-04T12:26:06.8079972Z  * [new branch]                gh/yushangdi/9/base     -> origin/gh/yushangdi/9/base
2025-12-04T12:26:06.8080041Z  * [new branch]                gh/yushangdi/9/head     -> origin/gh/yushangdi/9/head
2025-12-04T12:26:06.8080111Z  * [new branch]                gh/yushangdi/9/orig     -> origin/gh/yushangdi/9/orig
2025-12-04T12:26:06.8080181Z  * [new branch]                gh/zklaus/19/base       -> origin/gh/zklaus/19/base
2025-12-04T12:26:06.8080251Z  * [new branch]                gh/zklaus/19/head       -> origin/gh/zklaus/19/head
2025-12-04T12:26:06.8080321Z  * [new branch]                gh/zklaus/19/orig       -> origin/gh/zklaus/19/orig
2025-12-04T12:26:06.8080389Z  * [new branch]                gh/zklaus/20/base       -> origin/gh/zklaus/20/base
2025-12-04T12:26:06.8080456Z  * [new branch]                gh/zklaus/20/head       -> origin/gh/zklaus/20/head
2025-12-04T12:26:06.8080523Z  * [new branch]                gh/zklaus/20/orig       -> origin/gh/zklaus/20/orig
2025-12-04T12:26:06.8080589Z  * [new branch]                gh/zklaus/21/base       -> origin/gh/zklaus/21/base
2025-12-04T12:26:06.8080654Z  * [new branch]                gh/zklaus/21/head       -> origin/gh/zklaus/21/head
2025-12-04T12:26:06.8080727Z  * [new branch]                gh/zklaus/21/orig       -> origin/gh/zklaus/21/orig
2025-12-04T12:26:06.8080830Z  * [new branch]                gh/zklaus/22/base       -> origin/gh/zklaus/22/base
2025-12-04T12:26:06.8080898Z  * [new branch]                gh/zklaus/22/head       -> origin/gh/zklaus/22/head
2025-12-04T12:26:06.8080965Z  * [new branch]                gh/zklaus/22/orig       -> origin/gh/zklaus/22/orig
2025-12-04T12:26:06.8081060Z  * [new branch]                gh/zklaus/23/base       -> origin/gh/zklaus/23/base
2025-12-04T12:26:06.8081129Z  * [new branch]                gh/zklaus/23/head       -> origin/gh/zklaus/23/head
2025-12-04T12:26:06.8081196Z  * [new branch]                gh/zklaus/23/orig       -> origin/gh/zklaus/23/orig
2025-12-04T12:26:06.8081263Z  * [new branch]                gh/zklaus/24/base       -> origin/gh/zklaus/24/base
2025-12-04T12:26:06.8081330Z  * [new branch]                gh/zklaus/24/head       -> origin/gh/zklaus/24/head
2025-12-04T12:26:06.8081396Z  * [new branch]                gh/zklaus/24/orig       -> origin/gh/zklaus/24/orig
2025-12-04T12:26:06.8081468Z  * [new branch]                gh/zou3519/1197/base    -> origin/gh/zou3519/1197/base
2025-12-04T12:26:06.8081540Z  * [new branch]                gh/zou3519/1197/head    -> origin/gh/zou3519/1197/head
2025-12-04T12:26:06.8081613Z  * [new branch]                gh/zou3519/1197/orig    -> origin/gh/zou3519/1197/orig
2025-12-04T12:26:06.8081682Z  * [new branch]                gh/zou3519/1199/base    -> origin/gh/zou3519/1199/base
2025-12-04T12:26:06.8081752Z  * [new branch]                gh/zou3519/1199/head    -> origin/gh/zou3519/1199/head
2025-12-04T12:26:06.8081823Z  * [new branch]                gh/zou3519/1199/orig    -> origin/gh/zou3519/1199/orig
2025-12-04T12:26:06.8081891Z  * [new branch]                gh/zou3519/1200/base    -> origin/gh/zou3519/1200/base
2025-12-04T12:26:06.8081960Z  * [new branch]                gh/zou3519/1200/head    -> origin/gh/zou3519/1200/head
2025-12-04T12:26:06.8082029Z  * [new branch]                gh/zou3519/1200/orig    -> origin/gh/zou3519/1200/orig
2025-12-04T12:26:06.8082100Z  * [new branch]                gh/zou3519/1201/base    -> origin/gh/zou3519/1201/base
2025-12-04T12:26:06.8082172Z  * [new branch]                gh/zou3519/1201/head    -> origin/gh/zou3519/1201/head
2025-12-04T12:26:06.8082242Z  * [new branch]                gh/zou3519/1201/orig    -> origin/gh/zou3519/1201/orig
2025-12-04T12:26:06.8082312Z  * [new branch]                gh/zou3519/1202/base    -> origin/gh/zou3519/1202/base
2025-12-04T12:26:06.8082382Z  * [new branch]                gh/zou3519/1202/head    -> origin/gh/zou3519/1202/head
2025-12-04T12:26:06.8082451Z  * [new branch]                gh/zou3519/1202/orig    -> origin/gh/zou3519/1202/orig
2025-12-04T12:26:06.8082521Z  * [new branch]                gh/zpcore/1/base        -> origin/gh/zpcore/1/base
2025-12-04T12:26:06.8082590Z  * [new branch]                gh/zpcore/1/head        -> origin/gh/zpcore/1/head
2025-12-04T12:26:06.8082658Z  * [new branch]                gh/zpcore/11/base       -> origin/gh/zpcore/11/base
2025-12-04T12:26:06.8082728Z  * [new branch]                gh/zpcore/11/head       -> origin/gh/zpcore/11/head
2025-12-04T12:26:06.8082797Z  * [new branch]                gh/zpcore/11/orig       -> origin/gh/zpcore/11/orig
2025-12-04T12:26:06.8082864Z  * [new branch]                gh/zpcore/12/base       -> origin/gh/zpcore/12/base
2025-12-04T12:26:06.8082933Z  * [new branch]                gh/zpcore/12/head       -> origin/gh/zpcore/12/head
2025-12-04T12:26:06.8083001Z  * [new branch]                gh/zpcore/12/orig       -> origin/gh/zpcore/12/orig
2025-12-04T12:26:06.8083069Z  * [new branch]                gh/zpcore/13/base       -> origin/gh/zpcore/13/base
2025-12-04T12:26:06.8083136Z  * [new branch]                gh/zpcore/13/head       -> origin/gh/zpcore/13/head
2025-12-04T12:26:06.8083204Z  * [new branch]                gh/zpcore/13/orig       -> origin/gh/zpcore/13/orig
2025-12-04T12:26:06.8083272Z  * [new branch]                gh/zpcore/14/base       -> origin/gh/zpcore/14/base
2025-12-04T12:26:06.8083365Z  * [new branch]                gh/zpcore/14/head       -> origin/gh/zpcore/14/head
2025-12-04T12:26:06.8083435Z  * [new branch]                gh/zpcore/14/orig       -> origin/gh/zpcore/14/orig
2025-12-04T12:26:06.8083501Z  * [new branch]                gh/zpcore/15/base       -> origin/gh/zpcore/15/base
2025-12-04T12:26:06.8083594Z  * [new branch]                gh/zpcore/15/head       -> origin/gh/zpcore/15/head
2025-12-04T12:26:06.8083663Z  * [new branch]                gh/zpcore/15/orig       -> origin/gh/zpcore/15/orig
2025-12-04T12:26:06.8083729Z  * [new branch]                gh/zpcore/2/base        -> origin/gh/zpcore/2/base
2025-12-04T12:26:06.8083800Z  * [new branch]                gh/zpcore/2/head        -> origin/gh/zpcore/2/head
2025-12-04T12:26:06.8083867Z  * [new branch]                gh/zpcore/21/base       -> origin/gh/zpcore/21/base
2025-12-04T12:26:06.8083933Z  * [new branch]                gh/zpcore/21/head       -> origin/gh/zpcore/21/head
2025-12-04T12:26:06.8084000Z  * [new branch]                gh/zpcore/21/orig       -> origin/gh/zpcore/21/orig
2025-12-04T12:26:06.8084068Z  * [new branch]                gh/zpcore/22/base       -> origin/gh/zpcore/22/base
2025-12-04T12:26:06.8084135Z  * [new branch]                gh/zpcore/22/head       -> origin/gh/zpcore/22/head
2025-12-04T12:26:06.8084205Z  * [new branch]                gh/zpcore/22/orig       -> origin/gh/zpcore/22/orig
2025-12-04T12:26:06.8084271Z  * [new branch]                gh/zpcore/23/base       -> origin/gh/zpcore/23/base
2025-12-04T12:26:06.8084337Z  * [new branch]                gh/zpcore/23/head       -> origin/gh/zpcore/23/head
2025-12-04T12:26:06.8084403Z  * [new branch]                gh/zpcore/23/orig       -> origin/gh/zpcore/23/orig
2025-12-04T12:26:06.8084471Z  * [new branch]                gh/zpcore/24/base       -> origin/gh/zpcore/24/base
2025-12-04T12:26:06.8084537Z  * [new branch]                gh/zpcore/24/head       -> origin/gh/zpcore/24/head
2025-12-04T12:26:06.8084608Z  * [new branch]                gh/zpcore/24/orig       -> origin/gh/zpcore/24/orig
2025-12-04T12:26:06.8084677Z  * [new branch]                gh/zpcore/25/base       -> origin/gh/zpcore/25/base
2025-12-04T12:26:06.8084744Z  * [new branch]                gh/zpcore/25/head       -> origin/gh/zpcore/25/head
2025-12-04T12:26:06.8084813Z  * [new branch]                gh/zpcore/25/orig       -> origin/gh/zpcore/25/orig
2025-12-04T12:26:06.8084880Z  * [new branch]                gh/zpcore/26/base       -> origin/gh/zpcore/26/base
2025-12-04T12:26:06.8084947Z  * [new branch]                gh/zpcore/26/head       -> origin/gh/zpcore/26/head
2025-12-04T12:26:06.8085014Z  * [new branch]                gh/zpcore/26/orig       -> origin/gh/zpcore/26/orig
2025-12-04T12:26:06.8085080Z  * [new branch]                gh/zpcore/27/base       -> origin/gh/zpcore/27/base
2025-12-04T12:26:06.8085146Z  * [new branch]                gh/zpcore/27/head       -> origin/gh/zpcore/27/head
2025-12-04T12:26:06.8085213Z  * [new branch]                gh/zpcore/27/orig       -> origin/gh/zpcore/27/orig
2025-12-04T12:26:06.8085280Z  * [new branch]                gh/zpcore/28/base       -> origin/gh/zpcore/28/base
2025-12-04T12:26:06.8085349Z  * [new branch]                gh/zpcore/28/head       -> origin/gh/zpcore/28/head
2025-12-04T12:26:06.8085417Z  * [new branch]                gh/zpcore/28/orig       -> origin/gh/zpcore/28/orig
2025-12-04T12:26:06.8085483Z  * [new branch]                gh/zpcore/3/base        -> origin/gh/zpcore/3/base
2025-12-04T12:26:06.8085552Z  * [new branch]                gh/zpcore/3/head        -> origin/gh/zpcore/3/head
2025-12-04T12:26:06.8085618Z  * [new branch]                gh/zpcore/4/base        -> origin/gh/zpcore/4/base
2025-12-04T12:26:06.8085685Z  * [new branch]                gh/zpcore/4/head        -> origin/gh/zpcore/4/head
2025-12-04T12:26:06.8085752Z  * [new branch]                gh/zpcore/5/base        -> origin/gh/zpcore/5/base
2025-12-04T12:26:06.8085819Z  * [new branch]                gh/zpcore/5/head        -> origin/gh/zpcore/5/head
2025-12-04T12:26:06.8085906Z  * [new branch]                gh/zpcore/6/base        -> origin/gh/zpcore/6/base
2025-12-04T12:26:06.8085975Z  * [new branch]                gh/zpcore/6/head        -> origin/gh/zpcore/6/head
2025-12-04T12:26:06.8086041Z  * [new branch]                gh/zpcore/7/base        -> origin/gh/zpcore/7/base
2025-12-04T12:26:06.8086134Z  * [new branch]                gh/zpcore/7/head        -> origin/gh/zpcore/7/head
2025-12-04T12:26:06.8086203Z  * [new branch]                gh/zpcore/8/base        -> origin/gh/zpcore/8/base
2025-12-04T12:26:06.8086270Z  * [new branch]                gh/zpcore/8/head        -> origin/gh/zpcore/8/head
2025-12-04T12:26:06.8086340Z  * [new branch]                google-main             -> origin/google-main
2025-12-04T12:26:06.8086430Z  * [new branch]                guangyey/external_stream -> origin/guangyey/external_stream
2025-12-04T12:26:06.8086503Z  * [new branch]                guangyey/test_2025      -> origin/guangyey/test_2025
2025-12-04T12:26:06.8086648Z  * [new branch]                guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9
2025-12-04T12:26:06.8086806Z  * [new branch]                hameerabbasi/complex_tensor_subclass -> origin/hameerabbasi/complex_tensor_subclass
2025-12-04T12:26:06.8086951Z  * [new branch]                hameerabbasi/fix-ctensor-gradcheck-tests -> origin/hameerabbasi/fix-ctensor-gradcheck-tests
2025-12-04T12:26:06.8087061Z  * [new branch]                hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose
2025-12-04T12:26:06.8087126Z  * [new branch]                hc_baseline             -> origin/hc_baseline
2025-12-04T12:26:06.8087188Z  * [new branch]                hhh_rand                -> origin/hhh_rand
2025-12-04T12:26:06.8087252Z  * [new branch]                huba/f1                 -> origin/huba/f1
2025-12-04T12:26:06.8087444Z  * [new branch]                increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test -> origin/increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test
2025-12-04T12:26:06.8087508Z  * [new branch]                inlining                -> origin/inlining
2025-12-04T12:26:06.8087581Z  * [new branch]                inlining-ezyang         -> origin/inlining-ezyang
2025-12-04T12:26:06.8087666Z  * [new branch]                install-torchao-0.13.0  -> origin/install-torchao-0.13.0
2025-12-04T12:26:06.8087849Z  * [new branch]                instrument-trunk-pull-linux-with-job-test-filters -> origin/instrument-trunk-pull-linux-with-job-test-filters
2025-12-04T12:26:06.8087922Z  * [new branch]                invoke-subgraph         -> origin/invoke-subgraph
2025-12-04T12:26:06.8087989Z  * [new branch]                issue#58739             -> origin/issue#58739
2025-12-04T12:26:06.8088068Z  * [new branch]                jainapurva-patch-1      -> origin/jainapurva-patch-1
2025-12-04T12:26:06.8088130Z  * [new branch]                jathu/o3                -> origin/jathu/o3
2025-12-04T12:26:06.8088191Z  * [new branch]                jathu/sve               -> origin/jathu/sve
2025-12-04T12:26:06.8088315Z  * [new branch]                jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2
2025-12-04T12:26:06.8088423Z  * [new branch]                jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2
2025-12-04T12:26:06.8088537Z  * [new branch]                jiannanWang/memorysnapshot_filter -> origin/jiannanWang/memorysnapshot_filter
2025-12-04T12:26:06.8088649Z  * [new branch]                jiannanWang/profilerstepwarning -> origin/jiannanWang/profilerstepwarning
2025-12-04T12:26:06.8088738Z  * [new branch]                jithunnair-amd-patch-1  -> origin/jithunnair-amd-patch-1
2025-12-04T12:26:06.8088825Z  * [new branch]                jithunnair-amd-patch-10 -> origin/jithunnair-amd-patch-10
2025-12-04T12:26:06.8088909Z  * [new branch]                jithunnair-amd-patch-2  -> origin/jithunnair-amd-patch-2
2025-12-04T12:26:06.8088990Z  * [new branch]                jithunnair-amd-patch-3  -> origin/jithunnair-amd-patch-3
2025-12-04T12:26:06.8089110Z  * [new branch]                jithunnair-amd-patch-4  -> origin/jithunnair-amd-patch-4
2025-12-04T12:26:06.8089192Z  * [new branch]                jithunnair-amd-patch-5  -> origin/jithunnair-amd-patch-5
2025-12-04T12:26:06.8089301Z  * [new branch]                jithunnair-amd-patch-6  -> origin/jithunnair-amd-patch-6
2025-12-04T12:26:06.8089380Z  * [new branch]                jithunnair-amd-patch-7  -> origin/jithunnair-amd-patch-7
2025-12-04T12:26:06.8089459Z  * [new branch]                jithunnair-amd-patch-8  -> origin/jithunnair-amd-patch-8
2025-12-04T12:26:06.8089539Z  * [new branch]                jithunnair-amd-patch-9  -> origin/jithunnair-amd-patch-9
2025-12-04T12:26:06.8089616Z  * [new branch]                justinchu/native-qdq    -> origin/justinchu/native-qdq
2025-12-04T12:26:06.8089691Z  * [new branch]                kainan666/xlf_debug     -> origin/kainan666/xlf_debug
2025-12-04T12:26:06.8089754Z  * [new branch]                kainan_test             -> origin/kainan_test
2025-12-04T12:26:06.8089834Z  * [new branch]                larryliu0820-patch-1    -> origin/larryliu0820-patch-1
2025-12-04T12:26:06.8089941Z  * [new branch]                leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues
2025-12-04T12:26:06.8090047Z  * [new branch]                lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error
2025-12-04T12:26:06.8090126Z  * [new branch]                liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce
2025-12-04T12:26:06.8090233Z  * [new branch]                liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax
2025-12-04T12:26:06.8090314Z  * [new branch]                liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa
2025-12-04T12:26:06.8090384Z  * [new branch]                llama4-stable           -> origin/llama4-stable
2025-12-04T12:26:06.8090452Z  * [new branch]                lts/release/1.8         -> origin/lts/release/1.8
2025-12-04T12:26:06.8090525Z  * [new branch]                lucaskabela/#94773      -> origin/lucaskabela/#94773
2025-12-04T12:26:06.8090603Z  * [new branch]                lucaskabela/fix_164876  -> origin/lucaskabela/fix_164876
2025-12-04T12:26:06.8090688Z  * [new branch]                lucaskabela/flop_counter -> origin/lucaskabela/flop_counter
2025-12-04T12:26:06.8090786Z  * [new branch]                lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp
2025-12-04T12:26:06.8090894Z  * [new branch]                lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo
2025-12-04T12:26:06.8091019Z  * [new branch]                lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr
2025-12-04T12:26:06.8091133Z  * [new branch]                lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr
2025-12-04T12:26:06.8091268Z  * [new branch]                lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata
2025-12-04T12:26:06.8091350Z  * [new branch]                lucaskabela/rnn_decomp  -> origin/lucaskabela/rnn_decomp
2025-12-04T12:26:06.8091442Z  * [new branch]                lucaskabela/typing_backends -> origin/lucaskabela/typing_backends
2025-12-04T12:26:06.8091544Z  * [new branch]                lucaskabela/typing_ctx_manager -> origin/lucaskabela/typing_ctx_manager
2025-12-04T12:26:06.8091638Z  * [new branch]                lucaskabela/typing_nn_module -> origin/lucaskabela/typing_nn_module
2025-12-04T12:26:06.8091740Z  * [new branch]                lucaskabela/typing_user_defined -> origin/lucaskabela/typing_user_defined
2025-12-04T12:26:06.8091835Z  * [new branch]                lucaskabela/typing_variables -> origin/lucaskabela/typing_variables
2025-12-04T12:26:06.8091944Z  * [new branch]                lucaskabela/typing_variables_dicts -> origin/lucaskabela/typing_variables_dicts
2025-12-04T12:26:06.8092104Z  * [new branch]                lucaskabela/typing_variables_functions -> origin/lucaskabela/typing_variables_functions
2025-12-04T12:26:06.8092213Z  * [new branch]                lucaskabela/typing_variables_lists -> origin/lucaskabela/typing_variables_lists
2025-12-04T12:26:06.8092286Z  * [new branch]                lw/torch_box_by_ref     -> origin/lw/torch_box_by_ref
2025-12-04T12:26:06.8092372Z  * [new branch]                main                    -> origin/main
2025-12-04T12:26:06.8092442Z  * [new branch]                malfet-patch-1          -> origin/malfet-patch-1
2025-12-04T12:26:06.8092510Z  * [new branch]                malfet-patch-2          -> origin/malfet-patch-2
2025-12-04T12:26:06.8092577Z  * [new branch]                malfet-patch-3          -> origin/malfet-patch-3
2025-12-04T12:26:06.8092644Z  * [new branch]                malfet-patch-4          -> origin/malfet-patch-4
2025-12-04T12:26:06.8092709Z  * [new branch]                malfet-patch-5          -> origin/malfet-patch-5
2025-12-04T12:26:06.8092776Z  * [new branch]                malfet-patch-6          -> origin/malfet-patch-6
2025-12-04T12:26:06.8092845Z  * [new branch]                malfet-patch-7          -> origin/malfet-patch-7
2025-12-04T12:26:06.8092910Z  * [new branch]                malfet-patch-8          -> origin/malfet-patch-8
2025-12-04T12:26:06.8092987Z  * [new branch]                malfet/add-3.14-ci      -> origin/malfet/add-3.14-ci
2025-12-04T12:26:06.8093150Z  * [new branch]                malfet/be-do-not-make-typos-in-build-artifacts -> origin/malfet/be-do-not-make-typos-in-build-artifacts
2025-12-04T12:26:06.8093320Z  * [new branch]                malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch
2025-12-04T12:26:06.8093450Z  * [new branch]                malfet/be-remove-misisng-neon-headers -> origin/malfet/be-remove-misisng-neon-headers
2025-12-04T12:26:06.8093548Z  * [new branch]                malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im
2025-12-04T12:26:06.8093669Z  * [new branch]                manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe
2025-12-04T12:26:06.8093762Z  * [new branch]                manuel/inductor_link_openmp -> origin/manuel/inductor_link_openmp
2025-12-04T12:26:06.8093837Z  * [new branch]                masnesral/metaconda     -> origin/masnesral/metaconda
2025-12-04T12:26:06.8093917Z  * [new branch]                mem_profiler_flaky_fix  -> origin/mem_profiler_flaky_fix
2025-12-04T12:26:06.8093998Z  * [new branch]                mem_profiler_stack_trace -> origin/mem_profiler_stack_trace
2025-12-04T12:26:06.8094074Z  * [new branch]                memory_profiler_stack   -> origin/memory_profiler_stack
2025-12-04T12:26:06.8094149Z  * [new branch]                metascroy-patch-1       -> origin/metascroy-patch-1
2025-12-04T12:26:06.8094213Z  * [new branch]                mingw_posix             -> origin/mingw_posix
2025-12-04T12:26:06.8094287Z  * [new branch]                mlazos/S429861-debug    -> origin/mlazos/S429861-debug
2025-12-04T12:26:06.8094351Z  * [new branch]                mlazos/aa               -> origin/mlazos/aa
2025-12-04T12:26:06.8094414Z  * [new branch]                mlazos/acts             -> origin/mlazos/acts
2025-12-04T12:26:06.8094488Z  * [new branch]                mlazos/arg-renames      -> origin/mlazos/arg-renames
2025-12-04T12:26:06.8094570Z  * [new branch]                mlazos/bad-cudagraphs   -> origin/mlazos/bad-cudagraphs
2025-12-04T12:26:06.8094671Z  * [new branch]                mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks
2025-12-04T12:26:06.8094744Z  * [new branch]                mlazos/beta-tensor      -> origin/mlazos/beta-tensor
2025-12-04T12:26:06.8094810Z  * [new branch]                mlazos/buffers          -> origin/mlazos/buffers
2025-12-04T12:26:06.8094877Z  * [new branch]                mlazos/buffers2         -> origin/mlazos/buffers2
2025-12-04T12:26:06.8094944Z  * [new branch]                mlazos/buffers3         -> origin/mlazos/buffers3
2025-12-04T12:26:06.8095031Z  * [new branch]                mlazos/bwd              -> origin/mlazos/bwd
2025-12-04T12:26:06.8095104Z  * [new branch]                mlazos/combo-test       -> origin/mlazos/combo-test
2025-12-04T12:26:06.8095176Z  * [new branch]                mlazos/ctx-cleanup      -> origin/mlazos/ctx-cleanup
2025-12-04T12:26:06.8095277Z  * [new branch]                mlazos/cuda-cmd-log     -> origin/mlazos/cuda-cmd-log
2025-12-04T12:26:06.8095358Z  * [new branch]                mlazos/cudagraph-tests  -> origin/mlazos/cudagraph-tests
2025-12-04T12:26:06.8095462Z  * [new branch]                mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement
2025-12-04T12:26:06.8095538Z  * [new branch]                mlazos/cutlass-test     -> origin/mlazos/cutlass-test
2025-12-04T12:26:06.8095620Z  * [new branch]                mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug
2025-12-04T12:26:06.8095700Z  * [new branch]                mlazos/dataclass-proxy  -> origin/mlazos/dataclass-proxy
2025-12-04T12:26:06.8095770Z  * [new branch]                mlazos/dc-attrs         -> origin/mlazos/dc-attrs
2025-12-04T12:26:06.8095839Z  * [new branch]                mlazos/dc-helion        -> origin/mlazos/dc-helion
2025-12-04T12:26:06.8095907Z  * [new branch]                mlazos/dict-fix         -> origin/mlazos/dict-fix
2025-12-04T12:26:06.8095978Z  * [new branch]                mlazos/disable-tf       -> origin/mlazos/disable-tf
2025-12-04T12:26:06.8096045Z  * [new branch]                mlazos/dupe-fix         -> origin/mlazos/dupe-fix
2025-12-04T12:26:06.8096115Z  * [new branch]                mlazos/dyn-batch        -> origin/mlazos/dyn-batch
2025-12-04T12:26:06.8096177Z  * [new branch]                mlazos/evt              -> origin/mlazos/evt
2025-12-04T12:26:06.8096259Z  * [new branch]                mlazos/extract-examples -> origin/mlazos/extract-examples
2025-12-04T12:26:06.8096333Z  * [new branch]                mlazos/foreach-op       -> origin/mlazos/foreach-op
2025-12-04T12:26:06.8096397Z  * [new branch]                mlazos/fp8              -> origin/mlazos/fp8
2025-12-04T12:26:06.8096463Z  * [new branch]                mlazos/fp8-bias         -> origin/mlazos/fp8-bias
2025-12-04T12:26:06.8096547Z  * [new branch]                mlazos/fp8-bias-fusion  -> origin/mlazos/fp8-bias-fusion
2025-12-04T12:26:06.8096617Z  * [new branch]                mlazos/fp8-fixes        -> origin/mlazos/fp8-fixes
2025-12-04T12:26:06.8096683Z  * [new branch]                mlazos/freezing         -> origin/mlazos/freezing
2025-12-04T12:26:06.8096783Z  * [new branch]                mlazos/h-comp           -> origin/mlazos/h-comp
2025-12-04T12:26:06.8096852Z  * [new branch]                mlazos/h-comp2          -> origin/mlazos/h-comp2
2025-12-04T12:26:06.8096918Z  * [new branch]                mlazos/hash-hop         -> origin/mlazos/hash-hop
2025-12-04T12:26:06.8096980Z  * [new branch]                mlazos/hc               -> origin/mlazos/hc
2025-12-04T12:26:06.8097048Z  * [new branch]                mlazos/hc-cycles        -> origin/mlazos/hc-cycles
2025-12-04T12:26:06.8097117Z  * [new branch]                mlazos/hc-fixes         -> origin/mlazos/hc-fixes
2025-12-04T12:26:06.8097186Z  * [new branch]                mlazos/hc-fixes3        -> origin/mlazos/hc-fixes3
2025-12-04T12:26:06.8097256Z  * [new branch]                mlazos/hc-fixes4        -> origin/mlazos/hc-fixes4
2025-12-04T12:26:06.8097321Z  * [new branch]                mlazos/hc-hf            -> origin/mlazos/hc-hf
2025-12-04T12:26:06.8097385Z  * [new branch]                mlazos/hc-mut           -> origin/mlazos/hc-mut
2025-12-04T12:26:06.8097447Z  * [new branch]                mlazos/hc10             -> origin/mlazos/hc10
2025-12-04T12:26:06.8097509Z  * [new branch]                mlazos/hc11             -> origin/mlazos/hc11
2025-12-04T12:26:06.8097571Z  * [new branch]                mlazos/hc12             -> origin/mlazos/hc12
2025-12-04T12:26:06.8097632Z  * [new branch]                mlazos/hc13             -> origin/mlazos/hc13
2025-12-04T12:26:06.8097731Z  * [new branch]                mlazos/hc14             -> origin/mlazos/hc14
2025-12-04T12:26:06.8097793Z  * [new branch]                mlazos/hc15             -> origin/mlazos/hc15
2025-12-04T12:26:06.8097854Z  * [new branch]                mlazos/hc2              -> origin/mlazos/hc2
2025-12-04T12:26:06.8097954Z  * [new branch]                mlazos/hc4              -> origin/mlazos/hc4
2025-12-04T12:26:06.8098015Z  * [new branch]                mlazos/hc5              -> origin/mlazos/hc5
2025-12-04T12:26:06.8098075Z  * [new branch]                mlazos/hc6              -> origin/mlazos/hc6
2025-12-04T12:26:06.8098134Z  * [new branch]                mlazos/hc7              -> origin/mlazos/hc7
2025-12-04T12:26:06.8098193Z  * [new branch]                mlazos/hc8              -> origin/mlazos/hc8
2025-12-04T12:26:06.8098251Z  * [new branch]                mlazos/hc9              -> origin/mlazos/hc9
2025-12-04T12:26:06.8098325Z  * [new branch]                mlazos/hc_baseline2     -> origin/mlazos/hc_baseline2
2025-12-04T12:26:06.8098408Z  * [new branch]                mlazos/inductor-streams -> origin/mlazos/inductor-streams
2025-12-04T12:26:06.8098469Z  * [new branch]                mlazos/main             -> origin/mlazos/main
2025-12-04T12:26:06.8098533Z  * [new branch]                mlazos/mcg2             -> origin/mlazos/mcg2
2025-12-04T12:26:06.8098606Z  * [new branch]                mlazos/meta-guards      -> origin/mlazos/meta-guards
2025-12-04T12:26:06.8098709Z  * [new branch]                mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam
2025-12-04T12:26:06.8098807Z  * [new branch]                mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup
2025-12-04T12:26:06.8098873Z  * [new branch]                mlazos/mod-fix          -> origin/mlazos/mod-fix
2025-12-04T12:26:06.8098939Z  * [new branch]                mlazos/mode-fix         -> origin/mlazos/mode-fix
2025-12-04T12:26:06.8099005Z  * [new branch]                mlazos/offsets          -> origin/mlazos/offsets
2025-12-04T12:26:06.8099080Z  * [new branch]                mlazos/overguarding     -> origin/mlazos/overguarding
2025-12-04T12:26:06.8099155Z  * [new branch]                mlazos/proxy-ctors      -> origin/mlazos/proxy-ctors
2025-12-04T12:26:06.8099224Z  * [new branch]                mlazos/quant-fix        -> origin/mlazos/quant-fix
2025-12-04T12:26:06.8099294Z  * [new branch]                mlazos/resnet-fix       -> origin/mlazos/resnet-fix
2025-12-04T12:26:06.8099368Z  * [new branch]                mlazos/rm-buf-names     -> origin/mlazos/rm-buf-names
2025-12-04T12:26:06.8099433Z  * [new branch]                mlazos/rm-code          -> origin/mlazos/rm-code
2025-12-04T12:26:06.8099499Z  * [new branch]                mlazos/rm-spam          -> origin/mlazos/rm-spam
2025-12-04T12:26:06.8099562Z  * [new branch]                mlazos/rtp              -> origin/mlazos/rtp
2025-12-04T12:26:06.8099640Z  * [new branch]                mlazos/static-idx-dbg   -> origin/mlazos/static-idx-dbg
2025-12-04T12:26:06.8099727Z  * [new branch]                mlazos/static-inputs-log -> origin/mlazos/static-inputs-log
2025-12-04T12:26:06.8099792Z  * [new branch]                mlazos/stests           -> origin/mlazos/stests
2025-12-04T12:26:06.8099864Z  * [new branch]                mlazos/stream-ops       -> origin/mlazos/stream-ops
2025-12-04T12:26:06.8099930Z  * [new branch]                mlazos/td-fix2          -> origin/mlazos/td-fix2
2025-12-04T12:26:06.8100009Z  * [new branch]                mlazos/tensor-hasattr2  -> origin/mlazos/tensor-hasattr2
2025-12-04T12:26:06.8100070Z  * [new branch]                mlazos/test             -> origin/mlazos/test
2025-12-04T12:26:06.8100135Z  * [new branch]                mlazos/tf-mode          -> origin/mlazos/tf-mode
2025-12-04T12:26:06.8100215Z  * [new branch]                mlazos/tf-mode-backup2  -> origin/mlazos/tf-mode-backup2
2025-12-04T12:26:06.8100291Z  * [new branch]                mlazos/tf-mode-reland   -> origin/mlazos/tf-mode-reland
2025-12-04T12:26:06.8100390Z  * [new branch]                mlazos/tf-mode-reland2  -> origin/mlazos/tf-mode-reland2
2025-12-04T12:26:06.8100467Z  * [new branch]                mlazos/tf-mode-reland3  -> origin/mlazos/tf-mode-reland3
2025-12-04T12:26:06.8100579Z  * [new branch]                mlazos/triton-no-epi    -> origin/mlazos/triton-no-epi
2025-12-04T12:26:06.8100651Z  * [new branch]                mlazos/tune-proto       -> origin/mlazos/tune-proto
2025-12-04T12:26:06.8100724Z  * [new branch]                mlazos/tuple-fixes      -> origin/mlazos/tuple-fixes
2025-12-04T12:26:06.8100798Z  * [new branch]                mlazos/tuple-fixes2     -> origin/mlazos/tuple-fixes2
2025-12-04T12:26:06.8100876Z  * [new branch]                mlazos/tuple-handling   -> origin/mlazos/tuple-handling
2025-12-04T12:26:06.8100957Z  * [new branch]                mlazos/user-stream-base -> origin/mlazos/user-stream-base
2025-12-04T12:26:06.8101030Z  * [new branch]                mlazos/user-streams     -> origin/mlazos/user-streams
2025-12-04T12:26:06.8101125Z  * [new branch]                mlazos/user-streams-backup -> origin/mlazos/user-streams-backup
2025-12-04T12:26:06.8101220Z  * [new branch]                mlazos/user-streams-backup2 -> origin/mlazos/user-streams-backup2
2025-12-04T12:26:06.8101289Z  * [new branch]                mlazos/vary-beta        -> origin/mlazos/vary-beta
2025-12-04T12:26:06.8101361Z  * [new branch]                mlazos/vary-beta2       -> origin/mlazos/vary-beta2
2025-12-04T12:26:06.8101433Z  * [new branch]                mlazos/weird-perf1      -> origin/mlazos/weird-perf1
2025-12-04T12:26:06.8101505Z  * [new branch]                mm_out_dtype_compile    -> origin/mm_out_dtype_compile
2025-12-04T12:26:06.8101570Z  * [new branch]                module-shim             -> origin/module-shim
2025-12-04T12:26:06.8101631Z  * [new branch]                move_config             -> origin/move_config
2025-12-04T12:26:06.8101700Z  * [new branch]                msaroufim/reduce        -> origin/msaroufim/reduce
2025-12-04T12:26:06.8101772Z  * [new branch]                mtia/basic-cmake        -> origin/mtia/basic-cmake
2025-12-04T12:26:06.8101875Z  * [new branch]                mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape
2025-12-04T12:26:06.8101942Z  * [new branch]                my_varlen_backup        -> origin/my_varlen_backup
2025-12-04T12:26:06.8102017Z  * [new branch]                nativert_num_outputs    -> origin/nativert_num_outputs
2025-12-04T12:26:06.8102080Z  * [new branch]                new-codegen             -> origin/new-codegen
2025-12-04T12:26:06.8102145Z  * [new branch]                newtest-base            -> origin/newtest-base
2025-12-04T12:26:06.8102218Z  * [new branch]                ngimel/addmm_dtype      -> origin/ngimel/addmm_dtype
2025-12-04T12:26:06.8102285Z  * [new branch]                ngimel/div_inv          -> origin/ngimel/div_inv
2025-12-04T12:26:06.8102364Z  * [new branch]                ngimel/error_index_list -> origin/ngimel/error_index_list
2025-12-04T12:26:06.8102437Z  * [new branch]                ngimel/gather_grid      -> origin/ngimel/gather_grid
2025-12-04T12:26:06.8102525Z  * [new branch]                ngimel/gather_grid_release -> origin/ngimel/gather_grid_release
2025-12-04T12:26:06.8102592Z  * [new branch]                ngimel/gg_new           -> origin/ngimel/gg_new
2025-12-04T12:26:06.8102658Z  * [new branch]                ngimel/hostalloc        -> origin/ngimel/hostalloc
2025-12-04T12:26:06.8102726Z  * [new branch]                ngimel/storage_id       -> origin/ngimel/storage_id
2025-12-04T12:26:06.8102788Z  * [new branch]                nightly                 -> origin/nightly
2025-12-04T12:26:06.8102905Z  * [new branch]                nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check
2025-12-04T12:26:06.8103029Z  * [new branch]                nikitaved/addmm_epilogue_fusions_2d_bias -> origin/nikitaved/addmm_epilogue_fusions_2d_bias
2025-12-04T12:26:06.8103184Z  * [new branch]                nikitaved/addmm_epilogue_fusions_inductor -> origin/nikitaved/addmm_epilogue_fusions_inductor
2025-12-04T12:26:06.8103309Z  * [new branch]                nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch
2025-12-04T12:26:06.8103457Z  * [new branch]                nikitaved/grad_addmm_epilogue_fusions -> origin/nikitaved/grad_addmm_epilogue_fusions
2025-12-04T12:26:06.8103571Z  * [new branch]                nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index
2025-12-04T12:26:06.8103638Z  * [new branch]                nikitaved/test          -> origin/nikitaved/test
2025-12-04T12:26:06.8103767Z  * [new branch]                nmacchioni-perf-test-async-autotune -> origin/nmacchioni-perf-test-async-autotune
2025-12-04T12:26:06.8103846Z  * [new branch]                no_distributed_log_spew -> origin/no_distributed_log_spew
2025-12-04T12:26:06.8103910Z  * [new branch]                nofun-hack              -> origin/nofun-hack
2025-12-04T12:26:06.8103974Z  * [new branch]                norm_bench              -> origin/norm_bench
2025-12-04T12:26:06.8104049Z  * [new branch]                nullplay/fuse_matmul    -> origin/nullplay/fuse_matmul
2025-12-04T12:26:06.8104123Z  * [new branch]                nullplay_fuse_matmul    -> origin/nullplay_fuse_matmul
2025-12-04T12:26:06.8104192Z  * [new branch]                optimizer_test          -> origin/optimizer_test
2025-12-04T12:26:06.8104261Z  * [new branch]                orig/release/1.10       -> origin/orig/release/1.10
2025-12-04T12:26:06.8104329Z  * [new branch]                orig/release/1.11       -> origin/orig/release/1.11
2025-12-04T12:26:06.8104397Z  * [new branch]                orig/release/1.12       -> origin/orig/release/1.12
2025-12-04T12:26:06.8104464Z  * [new branch]                orig/release/1.13       -> origin/orig/release/1.13
2025-12-04T12:26:06.8104530Z  * [new branch]                orig/release/1.6        -> origin/orig/release/1.6
2025-12-04T12:26:06.8104598Z  * [new branch]                orig/release/1.7        -> origin/orig/release/1.7
2025-12-04T12:26:06.8104663Z  * [new branch]                orig/release/1.8        -> origin/orig/release/1.8
2025-12-04T12:26:06.8104729Z  * [new branch]                orig/release/1.9        -> origin/orig/release/1.9
2025-12-04T12:26:06.8104796Z  * [new branch]                orig/release/2.0        -> origin/orig/release/2.0
2025-12-04T12:26:06.8104860Z  * [new branch]                orig/release/2.1        -> origin/orig/release/2.1
2025-12-04T12:26:06.8104925Z  * [new branch]                orig/release/2.2        -> origin/orig/release/2.2
2025-12-04T12:26:06.8104989Z  * [new branch]                orig/release/2.3        -> origin/orig/release/2.3
2025-12-04T12:26:06.8105053Z  * [new branch]                orig/release/2.4        -> origin/orig/release/2.4
2025-12-04T12:26:06.8105118Z  * [new branch]                orig/release/2.5        -> origin/orig/release/2.5
2025-12-04T12:26:06.8105185Z  * [new branch]                orig/release/2.6        -> origin/orig/release/2.6
2025-12-04T12:26:06.8105250Z  * [new branch]                orig/release/2.7        -> origin/orig/release/2.7
2025-12-04T12:26:06.8105316Z  * [new branch]                orig/release/2.8        -> origin/orig/release/2.8
2025-12-04T12:26:06.8105382Z  * [new branch]                orig/release/2.9        -> origin/orig/release/2.9
2025-12-04T12:26:06.8105467Z  * [new branch]                origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base
2025-12-04T12:26:06.8105552Z  * [new branch]                origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig
2025-12-04T12:26:06.8105634Z  * [new branch]                origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig
2025-12-04T12:26:06.8105701Z  * [new branch]                oulgen-patch-1          -> origin/oulgen-patch-1
2025-12-04T12:26:06.8105770Z  * [new branch]                oulgen-patch-2          -> origin/oulgen-patch-2
2025-12-04T12:26:06.8105862Z  * [new branch]                oulgen-patch-3          -> origin/oulgen-patch-3
2025-12-04T12:26:06.8105927Z  * [new branch]                oulgen-patch-4          -> origin/oulgen-patch-4
2025-12-04T12:26:06.8105996Z  * [new branch]                padded-tensor           -> origin/padded-tensor
2025-12-04T12:26:06.8106079Z  * [new branch]                pca2                    -> origin/pca2
2025-12-04T12:26:06.8106152Z  * [new branch]                per_channel_backup      -> origin/per_channel_backup
2025-12-04T12:26:06.8106215Z  * [new branch]                perf_ops                -> origin/perf_ops
2025-12-04T12:26:06.8106280Z  * [new branch]                perf_ops_2_9            -> origin/perf_ops_2_9
2025-12-04T12:26:06.8106351Z  * [new branch]                pianpwk-patch-1         -> origin/pianpwk-patch-1
2025-12-04T12:26:06.8106440Z  * [new branch]                pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode
2025-12-04T12:26:06.8106552Z  * [new branch]                pianpwk/_debug_mode_for_triton_draft -> origin/pianpwk/_debug_mode_for_triton_draft
2025-12-04T12:26:06.8106655Z  * [new branch]                pianpwk/_debug_nn_module_compile -> origin/pianpwk/_debug_nn_module_compile
2025-12-04T12:26:06.8106785Z  * [new branch]                pianpwk/_draft_triton_11_3 -> origin/pianpwk/_draft_triton_11_3
2025-12-04T12:26:06.8106882Z  * [new branch]                pianpwk/_manual_bucket_draft -> origin/pianpwk/_manual_bucket_draft
2025-12-04T12:26:06.8106986Z  * [new branch]                pianpwk/_profile_w_dispatch_keys -> origin/pianpwk/_profile_w_dispatch_keys
2025-12-04T12:26:06.8107085Z  * [new branch]                pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode
2025-12-04T12:26:06.8107190Z  * [new branch]                pianpwk/_unbacked_local_shard_size -> origin/pianpwk/_unbacked_local_shard_size
2025-12-04T12:26:06.8107266Z  * [new branch]                pianpwk/anomaly_tb      -> origin/pianpwk/anomaly_tb
2025-12-04T12:26:06.8107349Z  * [new branch]                pianpwk/auto_fx_annotate -> origin/pianpwk/auto_fx_annotate
2025-12-04T12:26:06.8107463Z  * [new branch]                pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export
2025-12-04T12:26:06.8107552Z  * [new branch]                pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf
2025-12-04T12:26:06.8107651Z  * [new branch]                pianpwk/debug_fwd_stack_traces -> origin/pianpwk/debug_fwd_stack_traces
2025-12-04T12:26:06.8107736Z  * [new branch]                pianpwk/debug_hash_tensor -> origin/pianpwk/debug_hash_tensor
2025-12-04T12:26:06.8107828Z  * [new branch]                pianpwk/debug_mode_annotate -> origin/pianpwk/debug_mode_annotate
2025-12-04T12:26:06.8107918Z  * [new branch]                pianpwk/debug_mode_defaults -> origin/pianpwk/debug_mode_defaults
2025-12-04T12:26:06.8108000Z  * [new branch]                pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks
2025-12-04T12:26:06.8108109Z  * [new branch]                pianpwk/debug_mode_opcall_refactor -> origin/pianpwk/debug_mode_opcall_refactor
2025-12-04T12:26:06.8108197Z  * [new branch]                pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids
2025-12-04T12:26:06.8108282Z  * [new branch]                pianpwk/debug_mode_triton -> origin/pianpwk/debug_mode_triton
2025-12-04T12:26:06.8108380Z  * [new branch]                pianpwk/debug_show_stack_trace -> origin/pianpwk/debug_show_stack_trace
2025-12-04T12:26:06.8108481Z  * [new branch]                pianpwk/debug_wait_on_collective -> origin/pianpwk/debug_wait_on_collective
2025-12-04T12:26:06.8108580Z  * [new branch]                pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf
2025-12-04T12:26:06.8108706Z  * [new branch]                pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug
2025-12-04T12:26:06.8108813Z  * [new branch]                pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile
2025-12-04T12:26:06.8108952Z  * [new branch]                pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn
2025-12-04T12:26:06.8109068Z  * [new branch]                pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5
2025-12-04T12:26:06.8109195Z  * [new branch]                pianpwk/dtensor_custom_chunk -> origin/pianpwk/dtensor_custom_chunk
2025-12-04T12:26:06.8109302Z  * [new branch]                pianpwk/dtensor_unbacked_keypath -> origin/pianpwk/dtensor_unbacked_keypath
2025-12-04T12:26:06.8109383Z  * [new branch]                pianpwk/event_list_tree -> origin/pianpwk/event_list_tree
2025-12-04T12:26:06.8109466Z  * [new branch]                pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs
2025-12-04T12:26:06.8109544Z  * [new branch]                pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel
2025-12-04T12:26:06.8109649Z  * [new branch]                pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft
2025-12-04T12:26:06.8109761Z  * [new branch]                pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat
2025-12-04T12:26:06.8109877Z  * [new branch]                pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better
2025-12-04T12:26:06.8109963Z  * [new branch]                pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook
2025-12-04T12:26:06.8110071Z  * [new branch]                pianpwk/skip_python_keys_alternate -> origin/pianpwk/skip_python_keys_alternate
2025-12-04T12:26:06.8110176Z  * [new branch]                pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards
2025-12-04T12:26:06.8110262Z  * [new branch]                pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft
2025-12-04T12:26:06.8110343Z  * [new branch]                pianpwk/symint_one_hot  -> origin/pianpwk/symint_one_hot
2025-12-04T12:26:06.8110458Z  * [new branch]                pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false
2025-12-04T12:26:06.8110557Z  * [new branch]                pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap
2025-12-04T12:26:06.8110643Z  * [new branch]                pianpwk/try_dumb_stuff  -> origin/pianpwk/try_dumb_stuff
2025-12-04T12:26:06.8110724Z  * [new branch]                pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2
2025-12-04T12:26:06.8110816Z  * [new branch]                pianpwk/unbacked_dtensor_mm -> origin/pianpwk/unbacked_dtensor_mm
2025-12-04T12:26:06.8110914Z  * [new branch]                pianpwk/unbacked_tracing_12_2 -> origin/pianpwk/unbacked_tracing_12_2
2025-12-04T12:26:06.8110991Z  * [new branch]                pianpwk/user_symints    -> origin/pianpwk/user_symints
2025-12-04T12:26:06.8111069Z  * [new branch]                pianpwk/wan21_reshape   -> origin/pianpwk/wan21_reshape
2025-12-04T12:26:06.8111164Z  * [new branch]                piz/fix_partial_backward_1112 -> origin/piz/fix_partial_backward_1112
2025-12-04T12:26:06.8111240Z  * [new branch]                piz/prop_cache_clean    -> origin/piz/prop_cache_clean
2025-12-04T12:26:06.8111309Z  * [new branch]                pool-separate           -> origin/pool-separate
2025-12-04T12:26:06.8111373Z  * [new branch]                pr-156087               -> origin/pr-156087
2025-12-04T12:26:06.8111435Z  * [new branch]                pr/131860               -> origin/pr/131860
2025-12-04T12:26:06.8111505Z  * [new branch]                predispatch_to          -> origin/predispatch_to
2025-12-04T12:26:06.8111572Z  * [new branch]                protect-c17             -> origin/protect-c17
2025-12-04T12:26:06.8111638Z  * [new branch]                pt-opt-cuda3            -> origin/pt-opt-cuda3
2025-12-04T12:26:06.8111719Z  * [new branch]                python_compiled_autograd -> origin/python_compiled_autograd
2025-12-04T12:26:06.8111850Z  * [new branch]                q1l1/fix_device_moved_constant_type_unknown -> origin/q1l1/fix_device_moved_constant_type_unknown
2025-12-04T12:26:06.8112015Z  * [new branch]                q1l1/fix_wrong_default_type_for_kernel_call_args -> origin/q1l1/fix_wrong_default_type_for_kernel_call_args
2025-12-04T12:26:06.8112098Z  * [new branch]                qchip/export-D54134695  -> origin/qchip/export-D54134695
2025-12-04T12:26:06.8112207Z  * [new branch]                quote-pytest_cache      -> origin/quote-pytest_cache
2025-12-04T12:26:06.8112305Z  * [new branch]                reland-accgrad-stream-warn -> origin/reland-accgrad-stream-warn
2025-12-04T12:26:06.8112370Z  * [new branch]                release/1.10            -> origin/release/1.10
2025-12-04T12:26:06.8112433Z  * [new branch]                release/1.11            -> origin/release/1.11
2025-12-04T12:26:06.8112497Z  * [new branch]                release/1.12            -> origin/release/1.12
2025-12-04T12:26:06.8112558Z  * [new branch]                release/1.13            -> origin/release/1.13
2025-12-04T12:26:06.8112620Z  * [new branch]                release/1.4             -> origin/release/1.4
2025-12-04T12:26:06.8112688Z  * [new branch]                release/1.4.1           -> origin/release/1.4.1
2025-12-04T12:26:06.8112751Z  * [new branch]                release/1.5             -> origin/release/1.5
2025-12-04T12:26:06.8112813Z  * [new branch]                release/1.6             -> origin/release/1.6
2025-12-04T12:26:06.8112874Z  * [new branch]                release/1.7             -> origin/release/1.7
2025-12-04T12:26:06.8112934Z  * [new branch]                release/1.8             -> origin/release/1.8
2025-12-04T12:26:06.8112995Z  * [new branch]                release/1.9             -> origin/release/1.9
2025-12-04T12:26:06.8113057Z  * [new branch]                release/2.0             -> origin/release/2.0
2025-12-04T12:26:06.8113117Z  * [new branch]                release/2.1             -> origin/release/2.1
2025-12-04T12:26:06.8113178Z  * [new branch]                release/2.2             -> origin/release/2.2
2025-12-04T12:26:06.8113242Z  * [new branch]                release/2.3             -> origin/release/2.3
2025-12-04T12:26:06.8113303Z  * [new branch]                release/2.4             -> origin/release/2.4
2025-12-04T12:26:06.8113363Z  * [new branch]                release/2.5             -> origin/release/2.5
2025-12-04T12:26:06.8113427Z  * [new branch]                release/2.6             -> origin/release/2.6
2025-12-04T12:26:06.8113488Z  * [new branch]                release/2.7             -> origin/release/2.7
2025-12-04T12:26:06.8113547Z  * [new branch]                release/2.8             -> origin/release/2.8
2025-12-04T12:26:06.8113607Z  * [new branch]                release/2.9             -> origin/release/2.9
2025-12-04T12:26:06.8113671Z  * [new branch]                release_notes           -> origin/release_notes
2025-12-04T12:26:06.8113746Z  * [new branch]                remove_pyinterpreter    -> origin/remove_pyinterpreter
2025-12-04T12:26:06.8113875Z  * [new branch]                replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836
2025-12-04T12:26:06.8113997Z  * [new branch]                replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248
2025-12-04T12:26:06.8114117Z  * [new branch]                replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324
2025-12-04T12:26:06.8114238Z  * [new branch]                replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020
2025-12-04T12:26:06.8114370Z  * [new branch]                revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head
2025-12-04T12:26:06.8114482Z  * [new branch]                revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head
2025-12-04T12:26:06.8114586Z  * [new branch]                revert-152361-gh/fadara01/1/head -> origin/revert-152361-gh/fadara01/1/head
2025-12-04T12:26:06.8114689Z  * [new branch]                revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head
2025-12-04T12:26:06.8114891Z  * [new branch]                revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_
2025-12-04T12:26:06.8114989Z  * [new branch]                revert-hoo-invoke-subgraph -> origin/revert-hoo-invoke-subgraph
2025-12-04T12:26:06.8115112Z  * [new branch]                revert_always_build_distributed -> origin/revert_always_build_distributed
2025-12-04T12:26:06.8115181Z  * [new branch]                rms_norm_patch          -> origin/rms_norm_patch
2025-12-04T12:26:06.8115278Z  * [new branch]                ruisi/fix_all_to_all_estimation -> origin/ruisi/fix_all_to_all_estimation
2025-12-04T12:26:06.8115363Z  * [new branch]                ruisi/fix_comm_estimation -> origin/ruisi/fix_comm_estimation
2025-12-04T12:26:06.8115472Z  * [new branch]                ruisi/fix_dynamic_shape_estimation -> origin/ruisi/fix_dynamic_shape_estimation
2025-12-04T12:26:06.8115570Z  * [new branch]                ruisi/fix_llama3_autobucketing -> origin/ruisi/fix_llama3_autobucketing
2025-12-04T12:26:06.8115676Z  * [new branch]                ruisi/fix_manual_bucketing_ep_pass -> origin/ruisi/fix_manual_bucketing_ep_pass
2025-12-04T12:26:06.8115761Z  * [new branch]                ruisi/manual_bucket_pass -> origin/ruisi/manual_bucket_pass
2025-12-04T12:26:06.8115910Z  * [new branch]                ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures
2025-12-04T12:26:06.8115999Z  * [new branch]                ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var
2025-12-04T12:26:06.8116076Z  * [new branch]                rzou/faketensor_bench   -> origin/rzou/faketensor_bench
2025-12-04T12:26:06.8116137Z  * [new branch]                rzou/njt                -> origin/rzou/njt
2025-12-04T12:26:06.8116199Z  * [new branch]                rzou/pca                -> origin/rzou/pca
2025-12-04T12:26:06.8116265Z  * [new branch]                rzou/realprop           -> origin/rzou/realprop
2025-12-04T12:26:06.8116330Z  * [new branch]                samplevllm              -> origin/samplevllm
2025-12-04T12:26:06.8116501Z  * [new branch]                sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm
2025-12-04T12:26:06.8116594Z  * [new branch]                sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA
2025-12-04T12:26:06.8116708Z  * [new branch]                sapling-pr-archive-tushar00jain -> origin/sapling-pr-archive-tushar00jain
2025-12-04T12:26:06.8116804Z  * [new branch]                save                    -> origin/save
2025-12-04T12:26:06.8116867Z  * [new branch]                scaled_mm               -> origin/scaled_mm
2025-12-04T12:26:06.8116931Z  * [new branch]                scan_attempt            -> origin/scan_attempt
2025-12-04T12:26:06.8116993Z  * [new branch]                sdym/2.5.1              -> origin/sdym/2.5.1
2025-12-04T12:26:06.8117101Z  * [new branch]                sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix
2025-12-04T12:26:06.8117178Z  * [new branch]                shengf/fx-xform-perf    -> origin/shengf/fx-xform-perf
2025-12-04T12:26:06.8117255Z  * [new branch]                shoumikhin-patch-1      -> origin/shoumikhin-patch-1
2025-12-04T12:26:06.8117332Z  * [new branch]                solve-accuracy-fix      -> origin/solve-accuracy-fix
2025-12-04T12:26:06.8117413Z  * [new branch]                some_rocm_inductor_skips -> origin/some_rocm_inductor_skips
2025-12-04T12:26:06.8117494Z  * [new branch]                soulitzer/stash-tls-ac  -> origin/soulitzer/stash-tls-ac
2025-12-04T12:26:06.8117576Z  * [new branch]                sparse-mm-bf16-support  -> origin/sparse-mm-bf16-support
2025-12-04T12:26:06.8117650Z  * [new branch]                starterTaskUpdate       -> origin/starterTaskUpdate
2025-12-04T12:26:06.8117708Z  * [new branch]                suo                     -> origin/suo
2025-12-04T12:26:06.8117812Z  * [new branch]                sve-poc                 -> origin/sve-poc
2025-12-04T12:26:06.8117877Z  * [new branch]                switch-bn               -> origin/switch-bn
2025-12-04T12:26:06.8117970Z  * [new branch]                sy_annotation_in_autograd_hop -> origin/sy_annotation_in_autograd_hop
2025-12-04T12:26:06.8118076Z  * [new branch]                sy_aot_eager_record     -> origin/sy_aot_eager_record
2025-12-04T12:26:06.8118148Z  * [new branch]                sy_custom_bucketing     -> origin/sy_custom_bucketing
2025-12-04T12:26:06.8118216Z  * [new branch]                sy_debug_mode_test      -> origin/sy_debug_mode_test
2025-12-04T12:26:06.8118280Z  * [new branch]                sy_deserialize          -> origin/sy_deserialize
2025-12-04T12:26:06.8118347Z  * [new branch]                sy_dump_gm_code         -> origin/sy_dump_gm_code
2025-12-04T12:26:06.8118407Z  * [new branch]                sy_exp                  -> origin/sy_exp
2025-12-04T12:26:06.8118478Z  * [new branch]                sy_export_annotation    -> origin/sy_export_annotation
2025-12-04T12:26:06.8118550Z  * [new branch]                sy_invoke_subgraph      -> origin/sy_invoke_subgraph
2025-12-04T12:26:06.8118617Z  * [new branch]                sy_kernel_bw_name       -> origin/sy_kernel_bw_name
2025-12-04T12:26:06.8118682Z  * [new branch]                sy_multi_arch           -> origin/sy_multi_arch
2025-12-04T12:26:06.8118751Z  * [new branch]                sy_nn_module_stack      -> origin/sy_nn_module_stack
2025-12-04T12:26:06.8118821Z  * [new branch]                sy_original_dtensor     -> origin/sy_original_dtensor
2025-12-04T12:26:06.8118887Z  * [new branch]                sy_profiler_cia         -> origin/sy_profiler_cia
2025-12-04T12:26:06.8118953Z  * [new branch]                symm_mem_sync           -> origin/symm_mem_sync
2025-12-04T12:26:06.8119037Z  * [new branch]                sympy-bottleneck-repro  -> origin/sympy-bottleneck-repro
2025-12-04T12:26:06.8119115Z  * [new branch]                tensordict_integration  -> origin/tensordict_integration
2025-12-04T12:26:06.8119197Z  * [new branch]                test-move-conda-builds  -> origin/test-move-conda-builds
2025-12-04T12:26:06.8119259Z  * [new branch]                test-old                -> origin/test-old
2025-12-04T12:26:06.8119326Z  * [new branch]                test/bmm_heur           -> origin/test/bmm_heur
2025-12-04T12:26:06.8119423Z  * [new branch]                tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix
2025-12-04T12:26:06.8119534Z  * [new branch]                tianren/customOp_enable_max_autotune -> origin/tianren/customOp_enable_max_autotune
2025-12-04T12:26:06.8119615Z  * [new branch]                tianren/customOp_fusion -> origin/tianren/customOp_fusion
2025-12-04T12:26:06.8119741Z  * [new branch]                tianren/customop_collectiveop_benchmark -> origin/tianren/customop_collectiveop_benchmark
2025-12-04T12:26:06.8119876Z  * [new branch]                tianren/customop_collectiveop_benchmark_fix -> origin/tianren/customop_collectiveop_benchmark_fix
2025-12-04T12:26:06.8119981Z  * [new branch]                tianren/customop_dynamic_config -> origin/tianren/customop_dynamic_config
2025-12-04T12:26:06.8120073Z  * [new branch]                tianren/dynamic_range_input -> origin/tianren/dynamic_range_input
2025-12-04T12:26:06.8120174Z  * [new branch]                tianren/dynamic_range_input_fix -> origin/tianren/dynamic_range_input_fix
2025-12-04T12:26:06.8120281Z  * [new branch]                tianren/dynamic_range_input_merge -> origin/tianren/dynamic_range_input_merge
2025-12-04T12:26:06.8120382Z  * [new branch]                tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp
2025-12-04T12:26:06.8120461Z  * [new branch]                tianren/fx_codegen_dump -> origin/tianren/fx_codegen_dump
2025-12-04T12:26:06.8120545Z  * [new branch]                tianren/symmetric_memory -> origin/tianren/symmetric_memory
2025-12-04T12:26:06.8120610Z  * [new branch]                tianren/test            -> origin/tianren/test
2025-12-04T12:26:06.8120710Z  * [new branch]                tidy_performance_cyy    -> origin/tidy_performance_cyy
2025-12-04T12:26:06.8120770Z  * [new branch]                tmp                     -> origin/tmp
2025-12-04T12:26:06.8120859Z  * [new branch]                torchtitan_ep           -> origin/torchtitan_ep
2025-12-04T12:26:06.8120938Z  * [new branch]                torchtitan_integration  -> origin/torchtitan_integration
2025-12-04T12:26:06.8121022Z  * [new branch]                trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora
2025-12-04T12:26:06.8121104Z  * [new branch]                traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests
2025-12-04T12:26:06.8121175Z  * [new branch]                tree_loop_vec_base      -> origin/tree_loop_vec_base
2025-12-04T12:26:06.8121241Z  * [new branch]                triton_kernel           -> origin/triton_kernel
2025-12-04T12:26:06.8121303Z  * [new branch]                tt_pkg_1908             -> origin/tt_pkg_1908
2025-12-04T12:26:06.8121367Z  * [new branch]                type_dec                -> origin/type_dec
2025-12-04T12:26:06.8121460Z  * [new branch]                udate-sphinx-dependancies -> origin/udate-sphinx-dependancies
2025-12-04T12:26:06.8121600Z  * [new branch]                update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1
2025-12-04T12:26:06.8121736Z  * [new branch]                update-audio-commit-hash/19087141161-1916-1 -> origin/update-audio-commit-hash/19087141161-1916-1
2025-12-04T12:26:06.8121869Z  * [new branch]                update-audio-commit-hash/19250643381-1929-1 -> origin/update-audio-commit-hash/19250643381-1929-1
2025-12-04T12:26:06.8122001Z  * [new branch]                update-audio-commit-hash/19397724337-1935-1 -> origin/update-audio-commit-hash/19397724337-1935-1
2025-12-04T12:26:06.8122135Z  * [new branch]                update-audio-commit-hash/19555670148-1941-1 -> origin/update-audio-commit-hash/19555670148-1941-1
2025-12-04T12:26:06.8122268Z  * [new branch]                update-audio-commit-hash/19750627930-1946-1 -> origin/update-audio-commit-hash/19750627930-1946-1
2025-12-04T12:26:06.8122407Z  * [new branch]                update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2
2025-12-04T12:26:06.8122545Z  * [new branch]                update-vision-commit-hash/19087141161-1916-1 -> origin/update-vision-commit-hash/19087141161-1916-1
2025-12-04T12:26:06.8122680Z  * [new branch]                update-vision-commit-hash/19184897099-1925-1 -> origin/update-vision-commit-hash/19184897099-1925-1
2025-12-04T12:26:06.8122815Z  * [new branch]                update-vision-commit-hash/19250643381-1929-1 -> origin/update-vision-commit-hash/19250643381-1929-1
2025-12-04T12:26:06.8122949Z  * [new branch]                update-vision-commit-hash/19381328640-1934-1 -> origin/update-vision-commit-hash/19381328640-1934-1
2025-12-04T12:26:06.8123084Z  * [new branch]                update-vision-commit-hash/19485237164-1938-1 -> origin/update-vision-commit-hash/19485237164-1938-1
2025-12-04T12:26:06.8123218Z  * [new branch]                update-vllm-commit-hash/18451675449-1879-1 -> origin/update-vllm-commit-hash/18451675449-1879-1
2025-12-04T12:26:06.8123304Z  * [new branch]                update-vllm-dockerfile  -> origin/update-vllm-dockerfile
2025-12-04T12:26:06.8123429Z  * [new branch]                update-xla-commit-hash/19224287370-211-1 -> origin/update-xla-commit-hash/19224287370-211-1
2025-12-04T12:26:06.8123554Z  * [new branch]                update-xla-commit-hash/19422028566-212-1 -> origin/update-xla-commit-hash/19422028566-212-1
2025-12-04T12:26:06.8123676Z  * [new branch]                update-xla-commit-hash/19626841311-213-1 -> origin/update-xla-commit-hash/19626841311-213-1
2025-12-04T12:26:06.8123805Z  * [new branch]                update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388
2025-12-04T12:26:06.8123921Z  * [new branch]                update_operator_readme  -> origin/update_operator_readme
2025-12-04T12:26:06.8124011Z  * [new branch]                update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736
2025-12-04T12:26:06.8124100Z  * [new branch]                update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173
2025-12-04T12:26:06.8124208Z  * [new branch]                update_slow_tests_1762155677 -> origin/update_slow_tests_1762155677
2025-12-04T12:26:06.8124293Z  * [new branch]                update_slow_tests_1763365283 -> origin/update_slow_tests_1763365283
2025-12-04T12:26:06.8124392Z  * [new branch]                update_submodule_FBGEMM -> origin/update_submodule_FBGEMM
2025-12-04T12:26:06.8124470Z  * [new branch]                update_submodule_kineto -> origin/update_submodule_kineto
2025-12-04T12:26:06.8124562Z  * [new branch]                update_submodule_tensorpipe -> origin/update_submodule_tensorpipe
2025-12-04T12:26:06.8124664Z  * [new branch]                upload-tests-for-autorevert -> origin/upload-tests-for-autorevert
2025-12-04T12:26:06.8124726Z  * [new branch]                v0.1.2                  -> origin/v0.1.2
2025-12-04T12:26:06.8124787Z  * [new branch]                v1.0.1                  -> origin/v1.0.1
2025-12-04T12:26:06.8124848Z  * [new branch]                v1.0.3                  -> origin/v1.0.3
2025-12-04T12:26:06.8124906Z  * [new branch]                v1.1.0                  -> origin/v1.1.0
2025-12-04T12:26:06.8124962Z  * [new branch]                v1.2.0                  -> origin/v1.2.0
2025-12-04T12:26:06.8125019Z  * [new branch]                v1.3.0                  -> origin/v1.3.0
2025-12-04T12:26:06.8125075Z  * [new branch]                v1.3.1                  -> origin/v1.3.1
2025-12-04T12:26:06.8125140Z  * [new branch]                validate_fn             -> origin/validate_fn
2025-12-04T12:26:06.8125208Z  * [new branch]                validations_2.6         -> origin/validations_2.6
2025-12-04T12:26:06.8125276Z  * [new branch]                validations_2.8         -> origin/validations_2.8
2025-12-04T12:26:06.8125342Z  * [new branch]                varlen-api              -> origin/varlen-api
2025-12-04T12:26:06.8125417Z  * [new branch]                varlen-api-backup       -> origin/varlen-api-backup
2025-12-04T12:26:06.8125496Z  * [new branch]                varlen_batch_invariance -> origin/varlen_batch_invariance
2025-12-04T12:26:06.8125561Z  * [new branch]                viable/strict           -> origin/viable/strict
2025-12-04T12:26:06.8125678Z  * [new branch]                vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy
2025-12-04T12:26:06.8125742Z  * [new branch]                vllmbuildci             -> origin/vllmbuildci
2025-12-04T12:26:06.8125804Z  * [new branch]                vllmpin                 -> origin/vllmpin
2025-12-04T12:26:06.8125893Z  * [new branch]                vscode-recommend-pyrefly -> origin/vscode-recommend-pyrefly
2025-12-04T12:26:06.8125961Z  * [new branch]                wdvr-patch-1            -> origin/wdvr-patch-1
2025-12-04T12:26:06.8126026Z  * [new branch]                wdvr/iss_145259         -> origin/wdvr/iss_145259
2025-12-04T12:26:06.8126086Z  * [new branch]                whc/pei                 -> origin/whc/pei
2025-12-04T12:26:06.8126152Z  * [new branch]                whc/pp_fix              -> origin/whc/pp_fix
2025-12-04T12:26:06.8126217Z  * [new branch]                whc/sharding            -> origin/whc/sharding
2025-12-04T12:26:06.8126281Z  * [new branch]                whc/sharding2           -> origin/whc/sharding2
2025-12-04T12:26:06.8126343Z  * [new branch]                whc/uneven              -> origin/whc/uneven
2025-12-04T12:26:06.8126414Z  * [new branch]                whc/uneven-merge        -> origin/whc/uneven-merge
2025-12-04T12:26:06.8126477Z  * [new branch]                win_warnings            -> origin/win_warnings
2025-12-04T12:26:06.8126553Z  * [new branch]                windows_libtorch_free   -> origin/windows_libtorch_free
2025-12-04T12:26:06.8126644Z  * [new branch]                xmfan-war               -> origin/xmfan-war
2025-12-04T12:26:06.8126707Z  * [new branch]                xmfan/ca_0516           -> origin/xmfan/ca_0516
2025-12-04T12:26:06.8126818Z  * [new branch]                xmfan/ca_1051b93192     -> origin/xmfan/ca_1051b93192
2025-12-04T12:26:06.8127010Z  * [new branch]                xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8
2025-12-04T12:26:06.8127081Z  * [new branch]                xmfan/ca_5a2be192d1     -> origin/xmfan/ca_5a2be192d1
2025-12-04T12:26:06.8127151Z  * [new branch]                xmfan/ca_9d59b516e9     -> origin/xmfan/ca_9d59b516e9
2025-12-04T12:26:06.8127215Z  * [new branch]                xmfan/ca_apr8           -> origin/xmfan/ca_apr8
2025-12-04T12:26:06.8127279Z  * [new branch]                xmfan/ca_base           -> origin/xmfan/ca_base
2025-12-04T12:26:06.8127347Z  * [new branch]                xmfan/ca_dynamic        -> origin/xmfan/ca_dynamic
2025-12-04T12:26:06.8127416Z  * [new branch]                xmfan/ca_fix_dyn        -> origin/xmfan/ca_fix_dyn
2025-12-04T12:26:06.8127489Z  * [new branch]                xmfan/ca_fix_lowering   -> origin/xmfan/ca_fix_lowering
2025-12-04T12:26:06.8127567Z  * [new branch]                xmfan/ca_fix_polyfills  -> origin/xmfan/ca_fix_polyfills
2025-12-04T12:26:06.8127630Z  * [new branch]                xmfan/ca_jan3           -> origin/xmfan/ca_jan3
2025-12-04T12:26:06.8127694Z  * [new branch]                xmfan/ca_jun18          -> origin/xmfan/ca_jun18
2025-12-04T12:26:06.8127759Z  * [new branch]                xmfan/ca_jun24          -> origin/xmfan/ca_jun24
2025-12-04T12:26:06.8127824Z  * [new branch]                xmfan/ca_nested         -> origin/xmfan/ca_nested
2025-12-04T12:26:06.8127892Z  * [new branch]                xmfan/ca_overhead       -> origin/xmfan/ca_overhead
2025-12-04T12:26:06.8127986Z  * [new branch]                xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451
2025-12-04T12:26:06.8128055Z  * [new branch]                xmfan/cacu_jun18        -> origin/xmfan/cacu_jun18
2025-12-04T12:26:06.8128121Z  * [new branch]                xmfan/cacu_jun19        -> origin/xmfan/cacu_jun19
2025-12-04T12:26:06.8128189Z  * [new branch]                xmfan/cacu_jun4         -> origin/xmfan/cacu_jun4
2025-12-04T12:26:06.8128272Z  * [new branch]                xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape
2025-12-04T12:26:06.8128371Z  * [new branch]                xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough
2025-12-04T12:26:06.8128527Z  * [new branch]                xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9
2025-12-04T12:26:06.8128674Z  * [new branch]                xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9
2025-12-04T12:26:06.8128745Z  * [new branch]                xmfan/single_step       -> origin/xmfan/single_step
2025-12-04T12:26:06.8128810Z  * [new branch]                xmfan/sth_0829          -> origin/xmfan/sth_0829
2025-12-04T12:26:06.8128872Z  * [new branch]                xmfan/test              -> origin/xmfan/test
2025-12-04T12:26:06.8128963Z  * [new branch]                yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr
2025-12-04T12:26:06.8129040Z  * [new branch]                yguo/new_latest_changes -> origin/yguo/new_latest_changes
2025-12-04T12:26:06.8129135Z  * [new branch]                yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes
2025-12-04T12:26:06.8129204Z  * [new branch]                yiming/bootcamp         -> origin/yiming/bootcamp
2025-12-04T12:26:06.8129306Z  * [new branch]                yiming/run_with_start_end_rng_hop -> origin/yiming/run_with_start_end_rng_hop
2025-12-04T12:26:06.8129371Z  * [new branch]                yolo-llama3             -> origin/yolo-llama3
2025-12-04T12:26:06.8129483Z  * [new branch]                zainr/canary-test       -> origin/zainr/canary-test
2025-12-04T12:26:06.8129573Z  * [new branch]                zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners
2025-12-04T12:26:06.8129653Z  * [new branch]                zainr/pull-migration-c  -> origin/zainr/pull-migration-c
2025-12-04T12:26:06.8129743Z  * [new branch]                zainr/test2             -> origin/zainr/test2
2025-12-04T12:26:06.8129816Z  * [new branch]                zasdfgbnm-patch-3       -> origin/zasdfgbnm-patch-3
2025-12-04T12:26:06.8129875Z  * [new branch]                zb2p                    -> origin/zb2p
2025-12-04T12:26:06.8129960Z  * [new branch]                zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2
2025-12-04T12:26:06.8130047Z  * [new branch]                zhxchen17/ci/vllm_lora_oom -> origin/zhxchen17/ci/vllm_lora_oom
2025-12-04T12:26:06.8130150Z  * [new branch]                zhxchen17/ci/vllm_multimodal_oom -> origin/zhxchen17/ci/vllm_multimodal_oom
2025-12-04T12:26:06.8130227Z  * [new branch]                zhxchen17/ci/vllm_pin   -> origin/zhxchen17/ci/vllm_pin
2025-12-04T12:26:06.8130353Z  * [new branch]                zhxchen17/dynamo/unsafe_drop_all_guards -> origin/zhxchen17/dynamo/unsafe_drop_all_guards
2025-12-04T12:26:06.8130454Z  * [new branch]                zhxchen17/export/call_override -> origin/zhxchen17/export/call_override
2025-12-04T12:26:06.8130540Z  * [new branch]                zhxchen17/export/codemod1 -> origin/zhxchen17/export/codemod1
2025-12-04T12:26:06.8130629Z  * [new branch]                zhxchen17/export/ctx_return -> origin/zhxchen17/export/ctx_return
2025-12-04T12:26:06.8130759Z  * [new branch]                zhxchen17/export/disable_side_effect_warn -> origin/zhxchen17/export/disable_side_effect_warn
2025-12-04T12:26:06.8130858Z  * [new branch]                zhxchen17/export/pytree_check -> origin/zhxchen17/export/pytree_check
2025-12-04T12:26:06.8130945Z  * [new branch]                zhxchen17/precompile/aoti -> origin/zhxchen17/precompile/aoti
2025-12-04T12:26:06.8131045Z  * [new branch]                zhxchen17/precompile/globals -> origin/zhxchen17/precompile/globals
2025-12-04T12:26:06.8131162Z  * [new branch]                zhxchen17/precompile/inductor_guards -> origin/zhxchen17/precompile/inductor_guards
2025-12-04T12:26:06.8131238Z  * [new branch]                zhxchen17/scratch/0     -> origin/zhxchen17/scratch/0
2025-12-04T12:26:06.8131345Z  * [new branch]                zhxchen17/torch_export_api_update -> origin/zhxchen17/torch_export_api_update
2025-12-04T12:26:06.8131422Z  * [new branch]                zhxhcen17/moodycamel    -> origin/zhxhcen17/moodycamel
2025-12-04T12:26:06.8131496Z  * [new branch]                zxiiro/build-times      -> origin/zxiiro/build-times
2025-12-04T12:26:06.8131570Z  * [new branch]                zxiiro/c7i.2xlarge      -> origin/zxiiro/c7i.2xlarge
2025-12-04T12:26:06.8131649Z  * [new branch]                zxiiro/c7i.2xlarge.h100 -> origin/zxiiro/c7i.2xlarge.h100
2025-12-04T12:26:06.8131712Z  * [new branch]                zxiiro/main             -> origin/zxiiro/main
2025-12-04T12:26:06.8131777Z  * [new branch]                zxiiro/risc64           -> origin/zxiiro/risc64
2025-12-04T12:26:06.8131869Z  * [new branch]                zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc
2025-12-04T12:26:06.8131939Z  t [tag update]                ciflow/trunk/169385     -> ciflow/trunk/169385
2025-12-04T12:26:07.0002964Z [command]/usr/bin/git rev-parse --verify --quiet ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32^{object}
2025-12-04T12:26:07.0194305Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:07.0200851Z ##[endgroup]
2025-12-04T12:26:07.0201069Z ##[group]Determining the checkout info
2025-12-04T12:26:07.0202526Z ##[endgroup]
2025-12-04T12:26:07.0208288Z [command]/usr/bin/git sparse-checkout disable
2025-12-04T12:26:07.0289727Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-12-04T12:26:07.0313873Z ##[group]Checking out the ref
2025-12-04T12:26:07.0315994Z [command]/usr/bin/git checkout --progress --force ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:07.1154870Z Previous HEAD position was 685ba6bc0117 add back legalize_graph for BC reason (#169541)
2025-12-04T12:26:07.1160627Z HEAD is now at ffd9b0fb4355 Resolve collective autotuning test failure on arm (#168919)
2025-12-04T12:26:07.1242334Z ##[endgroup]
2025-12-04T12:26:07.1242522Z ##[group]Setting up auth for fetching submodules
2025-12-04T12:26:07.1249569Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-12-04T12:26:07.1286427Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-12-04T12:26:07.1307691Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-12-04T12:26:07.1329716Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-12-04T12:26:07.1346865Z ##[endgroup]
2025-12-04T12:26:07.1347159Z ##[group]Fetching submodules
2025-12-04T12:26:07.1348601Z [command]/usr/bin/git submodule sync --recursive
2025-12-04T12:26:07.1551355Z Synchronizing submodule url for 'android/libs/fbjni'
2025-12-04T12:26:07.1570598Z Synchronizing submodule url for 'third_party/FP16'
2025-12-04T12:26:07.1582691Z Synchronizing submodule url for 'third_party/FXdiv'
2025-12-04T12:26:07.1598648Z Synchronizing submodule url for 'third_party/NNPACK'
2025-12-04T12:26:07.1610320Z Synchronizing submodule url for 'third_party/NVTX'
2025-12-04T12:26:07.1621800Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:07.1633216Z Synchronizing submodule url for 'third_party/XNNPACK'
2025-12-04T12:26:07.1652937Z Synchronizing submodule url for 'third_party/aiter'
2025-12-04T12:26:07.1664765Z Synchronizing submodule url for 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:07.1681532Z Synchronizing submodule url for 'third_party/benchmark'
2025-12-04T12:26:07.1696909Z Synchronizing submodule url for 'third_party/composable_kernel'
2025-12-04T12:26:07.1711454Z Synchronizing submodule url for 'third_party/cpp-httplib'
2025-12-04T12:26:07.1722367Z Synchronizing submodule url for 'third_party/cpuinfo'
2025-12-04T12:26:07.1738586Z Synchronizing submodule url for 'third_party/cudnn_frontend'
2025-12-04T12:26:07.1749959Z Synchronizing submodule url for 'third_party/cutlass'
2025-12-04T12:26:07.1763258Z Synchronizing submodule url for 'third_party/fbgemm'
2025-12-04T12:26:07.1775743Z Synchronizing submodule url for 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:07.1795085Z Synchronizing submodule url for 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:07.1817407Z Synchronizing submodule url for 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:07.1828933Z Synchronizing submodule url for 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:07.1841044Z Synchronizing submodule url for 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:07.1859387Z Synchronizing submodule url for 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:07.1875539Z Synchronizing submodule url for 'third_party/fbgemm/external/json'
2025-12-04T12:26:07.1889765Z Synchronizing submodule url for 'third_party/flash-attention'
2025-12-04T12:26:07.1904816Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:07.1918121Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:07.1933661Z Synchronizing submodule url for 'third_party/flatbuffers'
2025-12-04T12:26:07.1945200Z Synchronizing submodule url for 'third_party/fmt'
2025-12-04T12:26:07.1955824Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:07.1966063Z Synchronizing submodule url for 'third_party/gloo'
2025-12-04T12:26:07.1975335Z Synchronizing submodule url for 'third_party/googletest'
2025-12-04T12:26:07.1984472Z Synchronizing submodule url for 'third_party/ideep'
2025-12-04T12:26:07.1995429Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:07.2008985Z Synchronizing submodule url for 'third_party/ittapi'
2025-12-04T12:26:07.2017807Z Synchronizing submodule url for 'third_party/kineto'
2025-12-04T12:26:07.2030402Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:07.2042891Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:07.2052902Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:07.2065275Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:07.2085570Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:07.2095970Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:07.2111264Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:07.2122347Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:07.2133390Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:07.2144508Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:07.2153824Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:07.2163315Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:07.2175014Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:07.2191591Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:07.2201475Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:07.2212690Z Synchronizing submodule url for 'third_party/kleidiai'
2025-12-04T12:26:07.2223139Z Synchronizing submodule url for 'third_party/mimalloc'
2025-12-04T12:26:07.2234556Z Synchronizing submodule url for 'third_party/nlohmann'
2025-12-04T12:26:07.2245145Z Synchronizing submodule url for 'third_party/onnx'
2025-12-04T12:26:07.2260015Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:07.2272604Z Synchronizing submodule url for 'third_party/opentelemetry-cpp'
2025-12-04T12:26:07.2284456Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:07.2294882Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:07.2305483Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:07.2315396Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:07.2326013Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:07.2337157Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:07.2345621Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:07.2365065Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:07.2384457Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:07.2396220Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:07.2418528Z Synchronizing submodule url for 'third_party/pocketfft'
2025-12-04T12:26:07.2428590Z Synchronizing submodule url for 'third_party/protobuf'
2025-12-04T12:26:07.2447025Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:07.2457879Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:07.2470320Z Synchronizing submodule url for 'third_party/psimd'
2025-12-04T12:26:07.2485769Z Synchronizing submodule url for 'third_party/pthreadpool'
2025-12-04T12:26:07.2501860Z Synchronizing submodule url for 'third_party/pybind11'
2025-12-04T12:26:07.2513740Z Synchronizing submodule url for 'third_party/python-peachpy'
2025-12-04T12:26:07.2526148Z Synchronizing submodule url for 'third_party/sleef'
2025-12-04T12:26:07.2536845Z Synchronizing submodule url for 'third_party/tensorpipe'
2025-12-04T12:26:07.2547289Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:07.2557026Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:07.2566889Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:07.2575819Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:07.2588793Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:07.2620382Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-12-04T12:26:07.2935482Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-12-04T12:26:07.3018319Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-12-04T12:26:07.3074266Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-12-04T12:26:07.3190934Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-12-04T12:26:07.3265707Z Submodule path 'third_party/NVTX': checked out '3ebbc93ded7285963bff932c678fa367eb393ba6'
2025-12-04T12:26:07.3333649Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1'
2025-12-04T12:26:07.8351358Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-12-04T12:26:07.8537953Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150'
2025-12-04T12:26:07.8731986Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf'
2025-12-04T12:26:07.8865111Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f'
2025-12-04T12:26:07.9080854Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977'
2025-12-04T12:26:07.9164820Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246'
2025-12-04T12:26:07.9815144Z Submodule path 'third_party/cpuinfo': checked out 'f858c30bcb16f8effd5ff46996f0514539e17abc'
2025-12-04T12:26:07.9907079Z Submodule path 'third_party/cudnn_frontend': checked out '0b1577c8c83401237d601d0d0db5210506705396'
2025-12-04T12:26:08.0036301Z Submodule path 'third_party/cutlass': checked out 'f88806b1e31dfa579842638740216dd41fc6c588'
2025-12-04T12:26:08.0773147Z Submodule path 'third_party/fbgemm': checked out 'c0b988d39a9e47c794d699f29930ed4d7c7e13a4'
2025-12-04T12:26:08.1107610Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea'
2025-12-04T12:26:08.2928763Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977'
2025-12-04T12:26:08.3595716Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349'
2025-12-04T12:26:08.4751168Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '98125ce499b0fdf7ffbe0e3052f5b8709f4840f8'
2025-12-04T12:26:08.4964047Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-12-04T12:26:08.5060519Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691'
2025-12-04T12:26:08.5636264Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03'
2025-12-04T12:26:08.5735216Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-12-04T12:26:08.5943319Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-12-04T12:26:08.6066675Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-12-04T12:26:08.6249355Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757'
2025-12-04T12:26:08.6411815Z Submodule path 'third_party/fmt': checked out '407c905e45ad75fc29bf0f9bb7c5c2fd3475976f'
2025-12-04T12:26:08.6624751Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-12-04T12:26:08.6744962Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341'
2025-12-04T12:26:08.6926220Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-12-04T12:26:08.7008038Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-12-04T12:26:09.0965667Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-12-04T12:26:09.1072203Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959'
2025-12-04T12:26:09.1178150Z Submodule path 'third_party/kineto': checked out '31f85df8fbd89c188f14ef10f1ec65379786b943'
2025-12-04T12:26:09.1263882Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1'
2025-12-04T12:26:09.1376316Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-12-04T12:26:09.1431967Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-12-04T12:26:09.1507255Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-12-04T12:26:09.1577968Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-12-04T12:26:09.1639531Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-12-04T12:26:09.1695197Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-12-04T12:26:09.1766044Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-12-04T12:26:09.1872741Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-12-04T12:26:09.1935221Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-12-04T12:26:09.1999428Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a'
2025-12-04T12:26:09.2076863Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159'
2025-12-04T12:26:09.2153378Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-12-04T12:26:09.2216088Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21'
2025-12-04T12:26:09.2281656Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-12-04T12:26:09.2355080Z Submodule path 'third_party/kleidiai': checked out 'd7770c89632329a9914ef1a90289917597639cbe'
2025-12-04T12:26:09.2442271Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e'
2025-12-04T12:26:09.2538684Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72'
2025-12-04T12:26:09.4338088Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83'
2025-12-04T12:26:09.4538006Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-12-04T12:26:09.4659837Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-12-04T12:26:09.4721222Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-12-04T12:26:09.4798878Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-12-04T12:26:09.4860022Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-12-04T12:26:09.4949228Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-12-04T12:26:09.5007900Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-12-04T12:26:09.5060282Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-12-04T12:26:09.5128635Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-12-04T12:26:09.5217260Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-12-04T12:26:09.5288208Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-12-04T12:26:09.5450774Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-12-04T12:26:09.5520825Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa'
2025-12-04T12:26:09.6826368Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-12-04T12:26:09.6935524Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-12-04T12:26:09.7136800Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-12-04T12:26:09.7220448Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-12-04T12:26:09.7319671Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-12-04T12:26:09.7524489Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8'
2025-12-04T12:26:09.7753835Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-12-04T12:26:09.8008128Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68'
2025-12-04T12:26:09.8130208Z Submodule path 'third_party/tensorpipe': checked out '2b4cd91092d335a697416b2a3cb398283246849d'
2025-12-04T12:26:09.8337790Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-12-04T12:26:09.8435508Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-12-04T12:26:09.8722666Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b'
2025-12-04T12:26:09.8867084Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-12-04T12:26:09.8926842Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-12-04T12:26:09.8956097Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-12-04T12:26:09.9165565Z Entering 'android/libs/fbjni'
2025-12-04T12:26:09.9188619Z Entering 'third_party/FP16'
2025-12-04T12:26:09.9210823Z Entering 'third_party/FXdiv'
2025-12-04T12:26:09.9231396Z Entering 'third_party/NNPACK'
2025-12-04T12:26:09.9252346Z Entering 'third_party/NVTX'
2025-12-04T12:26:09.9276176Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:09.9299517Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:09.9325014Z Entering 'third_party/aiter'
2025-12-04T12:26:09.9346046Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:09.9370744Z Entering 'third_party/benchmark'
2025-12-04T12:26:09.9390620Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:09.9413910Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:09.9438865Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:09.9459514Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:09.9478542Z Entering 'third_party/cutlass'
2025-12-04T12:26:09.9501323Z Entering 'third_party/fbgemm'
2025-12-04T12:26:09.9526730Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:09.9545324Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:09.9571181Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:09.9596622Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:09.9627118Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:09.9654223Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:09.9676804Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:09.9700791Z Entering 'third_party/flash-attention'
2025-12-04T12:26:09.9719700Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:09.9746961Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:09.9773135Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:09.9800130Z Entering 'third_party/fmt'
2025-12-04T12:26:09.9822289Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:09.9843944Z Entering 'third_party/gloo'
2025-12-04T12:26:09.9872450Z Entering 'third_party/googletest'
2025-12-04T12:26:09.9893807Z Entering 'third_party/ideep'
2025-12-04T12:26:09.9913929Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:09.9937335Z Entering 'third_party/ittapi'
2025-12-04T12:26:09.9962742Z Entering 'third_party/kineto'
2025-12-04T12:26:09.9991637Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:10.0020567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:10.0040561Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:10.0063431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:10.0085240Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:10.0107045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:10.0130424Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:10.0153585Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:10.0173884Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:10.0195447Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:10.0218019Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:10.0236802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:10.0256430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:10.0280963Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:10.0299681Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:10.0324004Z Entering 'third_party/kleidiai'
2025-12-04T12:26:10.0343857Z Entering 'third_party/mimalloc'
2025-12-04T12:26:10.0363193Z Entering 'third_party/nlohmann'
2025-12-04T12:26:10.0383321Z Entering 'third_party/onnx'
2025-12-04T12:26:10.0419552Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:10.0447883Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:10.0471946Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:10.0492755Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:10.0513155Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:10.0531724Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:10.0553471Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:10.0572420Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:10.0590948Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:10.0610510Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:10.0629764Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:10.0659945Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:10.0693398Z Entering 'third_party/pocketfft'
2025-12-04T12:26:10.0712252Z Entering 'third_party/protobuf'
2025-12-04T12:26:10.0734203Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:10.0752390Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:10.0776727Z Entering 'third_party/psimd'
2025-12-04T12:26:10.0801330Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:10.0827592Z Entering 'third_party/pybind11'
2025-12-04T12:26:10.0863456Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:10.0884419Z Entering 'third_party/sleef'
2025-12-04T12:26:10.0903952Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:10.0927126Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:10.0945698Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:10.0977650Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:10.1003632Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:10.1028726Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:10.1075973Z ##[endgroup]
2025-12-04T12:26:10.1076252Z ##[group]Persisting credentials for submodules
2025-12-04T12:26:10.1083581Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-12-04T12:26:10.1299762Z Entering 'android/libs/fbjni'
2025-12-04T12:26:10.1328952Z Entering 'third_party/FP16'
2025-12-04T12:26:10.1359252Z Entering 'third_party/FXdiv'
2025-12-04T12:26:10.1387370Z Entering 'third_party/NNPACK'
2025-12-04T12:26:10.1411201Z Entering 'third_party/NVTX'
2025-12-04T12:26:10.1431897Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:10.1460818Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:10.1491934Z Entering 'third_party/aiter'
2025-12-04T12:26:10.1519957Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:10.1558190Z Entering 'third_party/benchmark'
2025-12-04T12:26:10.1591784Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:10.1619841Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:10.1648282Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:10.1674657Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:10.1697759Z Entering 'third_party/cutlass'
2025-12-04T12:26:10.1722345Z Entering 'third_party/fbgemm'
2025-12-04T12:26:10.1752353Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:10.1779599Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:10.1819344Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:10.1842426Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:10.1867412Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:10.1892742Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:10.1919743Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:10.1952596Z Entering 'third_party/flash-attention'
2025-12-04T12:26:10.1977406Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:10.2006146Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:10.2036068Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:10.2067375Z Entering 'third_party/fmt'
2025-12-04T12:26:10.2092388Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:10.2125224Z Entering 'third_party/gloo'
2025-12-04T12:26:10.2148052Z Entering 'third_party/googletest'
2025-12-04T12:26:10.2173285Z Entering 'third_party/ideep'
2025-12-04T12:26:10.2197750Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:10.2228289Z Entering 'third_party/ittapi'
2025-12-04T12:26:10.2251106Z Entering 'third_party/kineto'
2025-12-04T12:26:10.2273658Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:10.2296492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:10.2317461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:10.2343562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:10.2366847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:10.2398184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:10.2431981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:10.2453493Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:10.2474410Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:10.2501393Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:10.2528564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:10.2553891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:10.2585383Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:10.2614598Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:10.2636512Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:10.2658747Z Entering 'third_party/kleidiai'
2025-12-04T12:26:10.2680282Z Entering 'third_party/mimalloc'
2025-12-04T12:26:10.2702608Z Entering 'third_party/nlohmann'
2025-12-04T12:26:10.2725668Z Entering 'third_party/onnx'
2025-12-04T12:26:10.2766659Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:10.2784296Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:10.2816423Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:10.2840204Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:10.2864572Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:10.2887010Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:10.2908942Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:10.2928411Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:10.2953632Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:10.2976881Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:10.2999486Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:10.3021864Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:10.3049698Z Entering 'third_party/pocketfft'
2025-12-04T12:26:10.3074859Z Entering 'third_party/protobuf'
2025-12-04T12:26:10.3102814Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:10.3126322Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:10.3152952Z Entering 'third_party/psimd'
2025-12-04T12:26:10.3174303Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:10.3202620Z Entering 'third_party/pybind11'
2025-12-04T12:26:10.3234400Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:10.3257658Z Entering 'third_party/sleef'
2025-12-04T12:26:10.3278751Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:10.3299027Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:10.3321995Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:10.3344053Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:10.3370071Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:10.3394225Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:10.3440636Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-12-04T12:26:10.3645840Z Entering 'android/libs/fbjni'
2025-12-04T12:26:10.3670184Z file:/home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-12-04T12:26:10.3679681Z Entering 'third_party/FP16'
2025-12-04T12:26:10.3709681Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-12-04T12:26:10.3723049Z Entering 'third_party/FXdiv'
2025-12-04T12:26:10.3749288Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-12-04T12:26:10.3763823Z Entering 'third_party/NNPACK'
2025-12-04T12:26:10.3790326Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-12-04T12:26:10.3801335Z Entering 'third_party/NVTX'
2025-12-04T12:26:10.3828538Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-12-04T12:26:10.3838915Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:10.3857936Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-12-04T12:26:10.3870355Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:10.4079649Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-12-04T12:26:10.4094082Z Entering 'third_party/aiter'
2025-12-04T12:26:10.4125594Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-12-04T12:26:10.4137864Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:10.4165299Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-12-04T12:26:10.4181709Z Entering 'third_party/benchmark'
2025-12-04T12:26:10.4209831Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:10.4220367Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:10.4245891Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-12-04T12:26:10.4269386Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:10.4289975Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-12-04T12:26:10.4299980Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:10.4328111Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-12-04T12:26:10.4338420Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:10.4367900Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-12-04T12:26:10.4380408Z Entering 'third_party/cutlass'
2025-12-04T12:26:10.4408233Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-12-04T12:26:10.4423424Z Entering 'third_party/fbgemm'
2025-12-04T12:26:10.4445705Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-12-04T12:26:10.4457611Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:10.4479878Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-12-04T12:26:10.4493780Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:10.4517028Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-12-04T12:26:10.4530773Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:10.4555229Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-12-04T12:26:10.4569277Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:10.4590978Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-12-04T12:26:10.4603994Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:10.4624031Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-12-04T12:26:10.4634751Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:10.4656929Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-12-04T12:26:10.4666506Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:10.4690172Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-12-04T12:26:10.4701775Z Entering 'third_party/flash-attention'
2025-12-04T12:26:10.4720405Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-12-04T12:26:10.4729200Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:10.4753851Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-12-04T12:26:10.4772666Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:10.4796313Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-12-04T12:26:10.4811080Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:10.4840158Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-12-04T12:26:10.4854781Z Entering 'third_party/fmt'
2025-12-04T12:26:10.4884941Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:10.4897487Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:10.4926090Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-12-04T12:26:10.4940214Z Entering 'third_party/gloo'
2025-12-04T12:26:10.4959861Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-12-04T12:26:10.4973643Z Entering 'third_party/googletest'
2025-12-04T12:26:10.5003038Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:10.5013811Z Entering 'third_party/ideep'
2025-12-04T12:26:10.5035835Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-12-04T12:26:10.5047720Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:10.5071826Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-12-04T12:26:10.5089699Z Entering 'third_party/ittapi'
2025-12-04T12:26:10.5111457Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-12-04T12:26:10.5120797Z Entering 'third_party/kineto'
2025-12-04T12:26:10.5142983Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-12-04T12:26:10.5158467Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:10.5179614Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-12-04T12:26:10.5192365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:10.5214639Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-12-04T12:26:10.5226210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:10.5254242Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-12-04T12:26:10.5264761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:10.5295063Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:10.5305550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:10.5328078Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-12-04T12:26:10.5337942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:10.5360179Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-12-04T12:26:10.5371676Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:10.5397005Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-12-04T12:26:10.5407720Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:10.5426627Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:10.5436235Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:10.5462866Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-12-04T12:26:10.5473521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:10.5498643Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-12-04T12:26:10.5508837Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:10.5527288Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T12:26:10.5536403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:10.5559656Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T12:26:10.5572994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:10.5598692Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T12:26:10.5611808Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:10.5641811Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:10.5652572Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:10.5673244Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:10.5688423Z Entering 'third_party/kleidiai'
2025-12-04T12:26:10.5711043Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-12-04T12:26:10.5721220Z Entering 'third_party/mimalloc'
2025-12-04T12:26:10.5741696Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-12-04T12:26:10.5754167Z Entering 'third_party/nlohmann'
2025-12-04T12:26:10.5780109Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-12-04T12:26:10.5791354Z Entering 'third_party/onnx'
2025-12-04T12:26:10.5814801Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-12-04T12:26:10.5830260Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:10.5862969Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:10.5877520Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:10.5905803Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-12-04T12:26:10.5916919Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:10.5950315Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:10.5960341Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:10.5990468Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:10.6000658Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:10.6026813Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-12-04T12:26:10.6037078Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:10.6073976Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-12-04T12:26:10.6083239Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:10.6105294Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-12-04T12:26:10.6117779Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:10.6137149Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-12-04T12:26:10.6146436Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:10.6169705Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T12:26:10.6178828Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:10.6197827Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T12:26:10.6208258Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:10.6225737Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T12:26:10.6238962Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:10.6267972Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-12-04T12:26:10.6289026Z Entering 'third_party/pocketfft'
2025-12-04T12:26:10.6309804Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-12-04T12:26:10.6323086Z Entering 'third_party/protobuf'
2025-12-04T12:26:10.6344405Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-12-04T12:26:10.6357440Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:10.6390818Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:10.6400481Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:10.6420434Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:10.6433974Z Entering 'third_party/psimd'
2025-12-04T12:26:10.6458667Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-12-04T12:26:10.6466941Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:10.6497834Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-12-04T12:26:10.6508035Z Entering 'third_party/pybind11'
2025-12-04T12:26:10.6528101Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:10.6537167Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:10.6560429Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-12-04T12:26:10.6574408Z Entering 'third_party/sleef'
2025-12-04T12:26:10.6596365Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-12-04T12:26:10.6605609Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:10.6625567Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-12-04T12:26:10.6633959Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:10.6653107Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:10.6662254Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:10.6684540Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-12-04T12:26:10.6692839Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:10.6719200Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-12-04T12:26:10.6732838Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:10.6751508Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:10.6762852Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:10.6784490Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-12-04T12:26:10.6979868Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-12-04T12:26:10.7160677Z Entering 'android/libs/fbjni'
2025-12-04T12:26:10.7190432Z Entering 'third_party/FP16'
2025-12-04T12:26:10.7215084Z Entering 'third_party/FXdiv'
2025-12-04T12:26:10.7235513Z Entering 'third_party/NNPACK'
2025-12-04T12:26:10.7253934Z Entering 'third_party/NVTX'
2025-12-04T12:26:10.7272487Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:10.7290371Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:10.7316054Z Entering 'third_party/aiter'
2025-12-04T12:26:10.7336835Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:10.7369055Z Entering 'third_party/benchmark'
2025-12-04T12:26:10.7400105Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:10.7423272Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:10.7443414Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:10.7462155Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:10.7485829Z Entering 'third_party/cutlass'
2025-12-04T12:26:10.7511108Z Entering 'third_party/fbgemm'
2025-12-04T12:26:10.7530933Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:10.7562203Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:10.7589961Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:10.7611330Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:10.7632361Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:10.7649478Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:10.7668692Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:10.7691948Z Entering 'third_party/flash-attention'
2025-12-04T12:26:10.7719708Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:10.7741814Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:10.7763646Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:10.7782479Z Entering 'third_party/fmt'
2025-12-04T12:26:10.7801906Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:10.7820584Z Entering 'third_party/gloo'
2025-12-04T12:26:10.7841961Z Entering 'third_party/googletest'
2025-12-04T12:26:10.7860780Z Entering 'third_party/ideep'
2025-12-04T12:26:10.7881368Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:10.7903124Z Entering 'third_party/ittapi'
2025-12-04T12:26:10.7931441Z Entering 'third_party/kineto'
2025-12-04T12:26:10.7953521Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:10.7983706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:10.8009854Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:10.8032389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:10.8050703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:10.8068467Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:10.8090160Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:10.8107392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:10.8127069Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:10.8152943Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:10.8171697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:10.8189009Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:10.8216418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:10.8239117Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:10.8258755Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:10.8283560Z Entering 'third_party/kleidiai'
2025-12-04T12:26:10.8306463Z Entering 'third_party/mimalloc'
2025-12-04T12:26:10.8336687Z Entering 'third_party/nlohmann'
2025-12-04T12:26:10.8363107Z Entering 'third_party/onnx'
2025-12-04T12:26:10.8386644Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:10.8415984Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:10.8438789Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:10.8471259Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:10.8493117Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:10.8513989Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:10.8531862Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:10.8561264Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:10.8581709Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:10.8599512Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:10.8625209Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:10.8645233Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:10.8673457Z Entering 'third_party/pocketfft'
2025-12-04T12:26:10.8697621Z Entering 'third_party/protobuf'
2025-12-04T12:26:10.8724275Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:10.8742875Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:10.8763282Z Entering 'third_party/psimd'
2025-12-04T12:26:10.8785966Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:10.8807435Z Entering 'third_party/pybind11'
2025-12-04T12:26:10.8826639Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:10.8848371Z Entering 'third_party/sleef'
2025-12-04T12:26:10.8868128Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:10.8886435Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:10.8904022Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:10.8926930Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:10.8947066Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:10.8969426Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:10.9003559Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-12-04T12:26:10.9162640Z Entering 'android/libs/fbjni'
2025-12-04T12:26:10.9181962Z Entering 'third_party/FP16'
2025-12-04T12:26:10.9200724Z Entering 'third_party/FXdiv'
2025-12-04T12:26:10.9219253Z Entering 'third_party/NNPACK'
2025-12-04T12:26:10.9238993Z Entering 'third_party/NVTX'
2025-12-04T12:26:10.9261336Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:10.9290699Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:10.9316203Z Entering 'third_party/aiter'
2025-12-04T12:26:10.9340652Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:10.9369467Z Entering 'third_party/benchmark'
2025-12-04T12:26:10.9391408Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:10.9413374Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:10.9435987Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:10.9458968Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:10.9477844Z Entering 'third_party/cutlass'
2025-12-04T12:26:10.9502022Z Entering 'third_party/fbgemm'
2025-12-04T12:26:10.9521786Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:10.9549555Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:10.9571562Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:10.9595209Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:10.9619109Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:10.9636628Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:10.9653779Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:10.9677190Z Entering 'third_party/flash-attention'
2025-12-04T12:26:10.9696443Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:10.9719185Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:10.9749272Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:10.9769220Z Entering 'third_party/fmt'
2025-12-04T12:26:10.9788454Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:10.9807694Z Entering 'third_party/gloo'
2025-12-04T12:26:10.9826848Z Entering 'third_party/googletest'
2025-12-04T12:26:10.9848312Z Entering 'third_party/ideep'
2025-12-04T12:26:10.9868860Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:10.9895455Z Entering 'third_party/ittapi'
2025-12-04T12:26:10.9914221Z Entering 'third_party/kineto'
2025-12-04T12:26:10.9934581Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:10.9967202Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:11.0000060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:11.0020376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:11.0040663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:11.0067086Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:11.0088422Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:11.0108247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:11.0127485Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:11.0146001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:11.0163620Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:11.0181866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:11.0209818Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:11.0232713Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:11.0253502Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:11.0273603Z Entering 'third_party/kleidiai'
2025-12-04T12:26:11.0295149Z Entering 'third_party/mimalloc'
2025-12-04T12:26:11.0320549Z Entering 'third_party/nlohmann'
2025-12-04T12:26:11.0346342Z Entering 'third_party/onnx'
2025-12-04T12:26:11.0371354Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:11.0397085Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:11.0417124Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:11.0442877Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:11.0470794Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:11.0489628Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:11.0520490Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:11.0544397Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:11.0568383Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:11.0586062Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:11.0618867Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:11.0640514Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:11.0666716Z Entering 'third_party/pocketfft'
2025-12-04T12:26:11.0690227Z Entering 'third_party/protobuf'
2025-12-04T12:26:11.0713139Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:11.0734996Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:11.0756048Z Entering 'third_party/psimd'
2025-12-04T12:26:11.0777328Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:11.0808461Z Entering 'third_party/pybind11'
2025-12-04T12:26:11.0827529Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:11.0847865Z Entering 'third_party/sleef'
2025-12-04T12:26:11.0872537Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:11.0898287Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:11.0927129Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:11.0948212Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:11.0967285Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:11.0986128Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:11.1033793Z ##[endgroup]
2025-12-04T12:26:11.1319511Z [command]/usr/bin/git log -1 --format=%H
2025-12-04T12:26:11.1530398Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:11.1661353Z ##[group]Run actions/checkout@v4
2025-12-04T12:26:11.1661488Z with:
2025-12-04T12:26:11.1661608Z   ref: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:11.1661744Z   fetch-depth: 0
2025-12-04T12:26:11.1661843Z   submodules: recursive
2025-12-04T12:26:11.1661944Z   show-progress: false
2025-12-04T12:26:11.1662053Z   repository: pytorch/pytorch
2025-12-04T12:26:11.1662202Z   token: ***
2025-12-04T12:26:11.1662289Z   ssh-strict: true
2025-12-04T12:26:11.1662383Z   ssh-user: git
2025-12-04T12:26:11.1662476Z   persist-credentials: true
2025-12-04T12:26:11.1662598Z   clean: true
2025-12-04T12:26:11.1662699Z   sparse-checkout-cone-mode: true
2025-12-04T12:26:11.1662814Z   fetch-tags: false
2025-12-04T12:26:11.1662909Z   lfs: false
2025-12-04T12:26:11.1663003Z   set-safe-directory: true
2025-12-04T12:26:11.1663107Z env:
2025-12-04T12:26:11.1663199Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:11.1663317Z ##[endgroup]
2025-12-04T12:26:11.2127237Z Syncing repository: pytorch/pytorch
2025-12-04T12:26:11.2127878Z ##[group]Getting Git version info
2025-12-04T12:26:11.2128284Z Working directory is '/home/runner/_work/pytorch/pytorch'
2025-12-04T12:26:11.2139159Z [command]/usr/bin/git version
2025-12-04T12:26:11.2159136Z git version 2.52.0
2025-12-04T12:26:11.2169964Z ##[endgroup]
2025-12-04T12:26:11.2173901Z Copying '/home/runner/.gitconfig' to '/home/runner/_work/_temp/fb21070e-acec-40a5-bf03-3aad69f5f66a/.gitconfig'
2025-12-04T12:26:11.2179425Z Temporarily overriding HOME='/home/runner/_work/_temp/fb21070e-acec-40a5-bf03-3aad69f5f66a' before making global git config changes
2025-12-04T12:26:11.2179831Z Adding repository directory to the temporary git global config as a safe directory
2025-12-04T12:26:11.2186792Z [command]/usr/bin/git config --global --add safe.directory /home/runner/_work/pytorch/pytorch
2025-12-04T12:26:11.2209792Z [command]/usr/bin/git config --local --get remote.origin.url
2025-12-04T12:26:11.2224819Z https://github.com/pytorch/pytorch
2025-12-04T12:26:11.2235072Z ##[group]Removing previously created refs, to avoid conflicts
2025-12-04T12:26:11.2237795Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2025-12-04T12:26:11.2256067Z HEAD
2025-12-04T12:26:11.2287381Z ##[endgroup]
2025-12-04T12:26:11.2288830Z [command]/usr/bin/git submodule status
2025-12-04T12:26:11.2501439Z  7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe)
2025-12-04T12:26:11.2547279Z  4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081)
2025-12-04T12:26:11.2586961Z  b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327)
2025-12-04T12:26:11.2649654Z  c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0)
2025-12-04T12:26:11.2693974Z  3ebbc93ded7285963bff932c678fa367eb393ba6 third_party/NVTX (v3.1.0-313-g3ebbc93)
2025-12-04T12:26:11.2745706Z  1d8f600fd424278486eade7ed3e877c99f0846b1 third_party/VulkanMemoryAllocator (v2.1.0-982-g1d8f600)
2025-12-04T12:26:11.3051031Z  51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a0103656)
2025-12-04T12:26:11.3085049Z  01aae101b9e5e94d6c16a9514c9fb8df99c93150 third_party/aiter (v0.1.1-92-g01aae101)
2025-12-04T12:26:11.3109190Z  299e5928955cc62af9968370293b916f5130916f third_party/benchmark (v1.9.3)
2025-12-04T12:26:11.3168805Z  7fe50dc3da2069d6645d9deb8c017a876472a977 third_party/composable_kernel (rocm-6.4.3-459-g7fe50dc3d)
2025-12-04T12:26:11.3253215Z  89c932f313c6437c38f2982869beacc89c2f2246 third_party/cpp-httplib (v0.26.0)
2025-12-04T12:26:11.3341918Z  f858c30bcb16f8effd5ff46996f0514539e17abc third_party/cpuinfo (f858c30)
2025-12-04T12:26:11.3365882Z  0b1577c8c83401237d601d0d0db5210506705396 third_party/cudnn_frontend (v0.5-61-g0b1577c)
2025-12-04T12:26:11.3435775Z  f88806b1e31dfa579842638740216dd41fc6c588 third_party/cutlass (v4.3.1)
2025-12-04T12:26:11.3453096Z  c0b988d39a9e47c794d699f29930ed4d7c7e13a4 third_party/fbgemm (v1.4.0-rc1-2-gc0b988d39)
2025-12-04T12:26:11.3504396Z  979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4)
2025-12-04T12:26:11.3523305Z  a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757 third_party/flatbuffers (v24.12.23)
2025-12-04T12:26:11.3753140Z  407c905e45ad75fc29bf0f9bb7c5c2fd3475976f third_party/fmt (12.1.0)
2025-12-04T12:26:11.3823291Z  3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17)
2025-12-04T12:26:11.3912815Z  54cbae0d3a67fa890b4c3d9ee162b7860315e341 third_party/gloo (remotes/origin/gh/c-p-i-o/1/base-37-g54cbae0)
2025-12-04T12:26:11.4056389Z  52eb8108c5bdec04579160ae17225d66034bd723 third_party/googletest (release-1.8.0-3544-g52eb8108)
2025-12-04T12:26:11.4114211Z  719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1)
2025-12-04T12:26:11.4159931Z  dec1d23ca65ab069d225dfe40dea14f455170959 third_party/ittapi (v3.25.5)
2025-12-04T12:26:11.4290203Z  31f85df8fbd89c188f14ef10f1ec65379786b943 third_party/kineto (heads/main)
2025-12-04T12:26:11.4313281Z  d7770c89632329a9914ef1a90289917597639cbe third_party/kleidiai (v1.15.0)
2025-12-04T12:26:11.4332656Z  fbd8b99c2b828428947d70fdc046bb55609be93e third_party/mimalloc (v2.2.4)
2025-12-04T12:26:11.4353430Z  55f93686c01528224f448c19128836e7df245f72 third_party/nlohmann (v3.12.0)
2025-12-04T12:26:11.4557339Z  e709452ef2bbc1d113faf678c24e6d3467696e83 third_party/onnx (v1.18.0)
2025-12-04T12:26:11.4573210Z  a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2)
2025-12-04T12:26:11.4592850Z  0fa0ef591e38c2758e3184c6c23e497b9f732ffa third_party/pocketfft (release_for_eigen-40-g0fa0ef5)
2025-12-04T12:26:11.4795815Z  d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b)
2025-12-04T12:26:11.4844441Z  072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master)
2025-12-04T12:26:11.4890281Z  4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e)
2025-12-04T12:26:11.4906379Z  f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8 third_party/pybind11 (v3.0.1)
2025-12-04T12:26:11.4963175Z  f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated)
2025-12-04T12:26:11.5029550Z  5a1d179df9cf652951b59010a2d2075372d67f68 third_party/sleef (3.8)
2025-12-04T12:26:11.5079498Z  2b4cd91092d335a697416b2a3cb398283246849d third_party/tensorpipe (heads/main)
2025-12-04T12:26:11.5089938Z ##[group]Cleaning the repository
2025-12-04T12:26:11.5095179Z [command]/usr/bin/git clean -ffdx
2025-12-04T12:26:11.5208473Z [command]/usr/bin/git reset --hard HEAD
2025-12-04T12:26:11.6014443Z HEAD is now at ffd9b0fb4355 Resolve collective autotuning test failure on arm (#168919)
2025-12-04T12:26:11.6077409Z ##[endgroup]
2025-12-04T12:26:11.6080473Z ##[group]Disabling automatic garbage collection
2025-12-04T12:26:11.6086804Z [command]/usr/bin/git config --local gc.auto 0
2025-12-04T12:26:11.6113555Z ##[endgroup]
2025-12-04T12:26:11.6113872Z ##[group]Setting up auth
2025-12-04T12:26:11.6117095Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-12-04T12:26:11.6140634Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-12-04T12:26:11.6366828Z Entering 'android/libs/fbjni'
2025-12-04T12:26:11.6400859Z Entering 'third_party/FP16'
2025-12-04T12:26:11.6432866Z Entering 'third_party/FXdiv'
2025-12-04T12:26:11.6460945Z Entering 'third_party/NNPACK'
2025-12-04T12:26:11.6486709Z Entering 'third_party/NVTX'
2025-12-04T12:26:11.6516198Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:11.6539917Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:11.6564952Z Entering 'third_party/aiter'
2025-12-04T12:26:11.6590891Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:11.6625577Z Entering 'third_party/benchmark'
2025-12-04T12:26:11.6649899Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:11.6675336Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:11.6709472Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:11.6732524Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:11.6755092Z Entering 'third_party/cutlass'
2025-12-04T12:26:11.6781414Z Entering 'third_party/fbgemm'
2025-12-04T12:26:11.6808863Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:11.6837346Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:11.6869780Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:11.6891946Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:11.6919547Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:11.6944488Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:11.6964206Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:11.6988047Z Entering 'third_party/flash-attention'
2025-12-04T12:26:11.7011433Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:11.7039835Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:11.7071679Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:11.7097054Z Entering 'third_party/fmt'
2025-12-04T12:26:11.7120534Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:11.7149155Z Entering 'third_party/gloo'
2025-12-04T12:26:11.7172194Z Entering 'third_party/googletest'
2025-12-04T12:26:11.7193980Z Entering 'third_party/ideep'
2025-12-04T12:26:11.7214432Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:11.7240777Z Entering 'third_party/ittapi'
2025-12-04T12:26:11.7264393Z Entering 'third_party/kineto'
2025-12-04T12:26:11.7287394Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:11.7311950Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:11.7336286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:11.7364325Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:11.7390710Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:11.7421313Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:11.7447351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:11.7470073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:11.7494874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:11.7520037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:11.7547818Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:11.7575982Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:11.7597228Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:11.7626834Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:11.7657397Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:11.7688982Z Entering 'third_party/kleidiai'
2025-12-04T12:26:11.7716893Z Entering 'third_party/mimalloc'
2025-12-04T12:26:11.7739978Z Entering 'third_party/nlohmann'
2025-12-04T12:26:11.7760782Z Entering 'third_party/onnx'
2025-12-04T12:26:11.7794857Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:11.7822222Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:11.7845694Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:11.7884508Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:11.7893150Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:11.7917943Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:11.7940029Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:11.7965486Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:11.7990610Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:11.8010015Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:11.8034091Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:11.8059439Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:11.8088896Z Entering 'third_party/pocketfft'
2025-12-04T12:26:11.8111498Z Entering 'third_party/protobuf'
2025-12-04T12:26:11.8136721Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:11.8160156Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:11.8188727Z Entering 'third_party/psimd'
2025-12-04T12:26:11.8213328Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:11.8235285Z Entering 'third_party/pybind11'
2025-12-04T12:26:11.8258053Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:11.8280499Z Entering 'third_party/sleef'
2025-12-04T12:26:11.8301408Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:11.8325264Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:11.8358337Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:11.8384262Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:11.8408748Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:11.8431808Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:11.8483262Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-12-04T12:26:11.8501734Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8512265Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-12-04T12:26:11.8537407Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-12-04T12:26:11.8708181Z Entering 'android/libs/fbjni'
2025-12-04T12:26:11.8723351Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8747822Z Entering 'third_party/FP16'
2025-12-04T12:26:11.8761679Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8778683Z Entering 'third_party/FXdiv'
2025-12-04T12:26:11.8792948Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8808141Z Entering 'third_party/NNPACK'
2025-12-04T12:26:11.8821099Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8837529Z Entering 'third_party/NVTX'
2025-12-04T12:26:11.8851079Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8867367Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:11.8880982Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8900797Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:11.8914934Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8940042Z Entering 'third_party/aiter'
2025-12-04T12:26:11.8953197Z http.https://github.com/.extraheader
2025-12-04T12:26:11.8970285Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:11.8983251Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9004667Z Entering 'third_party/benchmark'
2025-12-04T12:26:11.9019906Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9037664Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:11.9049806Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9069531Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:11.9081125Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9097660Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:11.9108702Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9125957Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:11.9139240Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9155678Z Entering 'third_party/cutlass'
2025-12-04T12:26:11.9171275Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9200785Z Entering 'third_party/fbgemm'
2025-12-04T12:26:11.9214240Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9236630Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:11.9264757Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9284633Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:11.9296462Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9320145Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:11.9331514Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9350552Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:11.9361727Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9385811Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:11.9398521Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9423461Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:11.9439654Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9457439Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:11.9469576Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9490065Z Entering 'third_party/flash-attention'
2025-12-04T12:26:11.9503826Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9520460Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:11.9540155Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9562819Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:11.9579071Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9606190Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:11.9621516Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9640967Z Entering 'third_party/fmt'
2025-12-04T12:26:11.9657021Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9672053Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:11.9683105Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9701232Z Entering 'third_party/gloo'
2025-12-04T12:26:11.9715155Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9734331Z Entering 'third_party/googletest'
2025-12-04T12:26:11.9747371Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9767513Z Entering 'third_party/ideep'
2025-12-04T12:26:11.9779607Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9799821Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:11.9823701Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9845921Z Entering 'third_party/ittapi'
2025-12-04T12:26:11.9858560Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9881475Z Entering 'third_party/kineto'
2025-12-04T12:26:11.9895635Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9913811Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:11.9925556Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9942913Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:11.9957317Z http.https://github.com/.extraheader
2025-12-04T12:26:11.9982735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:12.0003091Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0022106Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:12.0038778Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0059182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:12.0074898Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0093207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:12.0111266Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0131792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:12.0144706Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0166869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:12.0182435Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0200005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:12.0216807Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0233442Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:12.0247116Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0265114Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:12.0280344Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0295221Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:12.0314221Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0333962Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:12.0349366Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0371370Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:12.0389365Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0406020Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:12.0427163Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0450270Z Entering 'third_party/kleidiai'
2025-12-04T12:26:12.0464634Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0484095Z Entering 'third_party/mimalloc'
2025-12-04T12:26:12.0499485Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0518104Z Entering 'third_party/nlohmann'
2025-12-04T12:26:12.0530726Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0555684Z Entering 'third_party/onnx'
2025-12-04T12:26:12.0573324Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0598488Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:12.0616300Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0642656Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:12.0658266Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0676280Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:12.0696350Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0721297Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:12.0735538Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0755327Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:12.0771827Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0788493Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:12.0806799Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0824235Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:12.0842732Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0868119Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:12.0884585Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0909547Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:12.0931747Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0952159Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:12.0968701Z http.https://github.com/.extraheader
2025-12-04T12:26:12.0986562Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:12.1006122Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1027761Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:12.1044517Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1078984Z Entering 'third_party/pocketfft'
2025-12-04T12:26:12.1096472Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1118316Z Entering 'third_party/protobuf'
2025-12-04T12:26:12.1131482Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1156045Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:12.1174456Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1191682Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:12.1204445Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1223483Z Entering 'third_party/psimd'
2025-12-04T12:26:12.1236372Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1254218Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:12.1271708Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1289614Z Entering 'third_party/pybind11'
2025-12-04T12:26:12.1304622Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1330473Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:12.1342810Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1368250Z Entering 'third_party/sleef'
2025-12-04T12:26:12.1383238Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1404733Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:12.1419645Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1443865Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:12.1461023Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1478221Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:12.1490567Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1504706Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:12.1516223Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1535304Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:12.1550900Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1570126Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:12.1584199Z http.https://github.com/.extraheader
2025-12-04T12:26:12.1629136Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.1651158Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
2025-12-04T12:26:12.1879179Z Entering 'android/libs/fbjni'
2025-12-04T12:26:12.1893883Z file:/home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-12-04T12:26:12.1905432Z Entering 'third_party/FP16'
2025-12-04T12:26:12.1918367Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-12-04T12:26:12.1928109Z Entering 'third_party/FXdiv'
2025-12-04T12:26:12.1941450Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-12-04T12:26:12.1953023Z Entering 'third_party/NNPACK'
2025-12-04T12:26:12.1970619Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-12-04T12:26:12.1979958Z Entering 'third_party/NVTX'
2025-12-04T12:26:12.1990545Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-12-04T12:26:12.2006260Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:12.2016897Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-12-04T12:26:12.2029383Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:12.2042627Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-12-04T12:26:12.2059238Z Entering 'third_party/aiter'
2025-12-04T12:26:12.2069182Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-12-04T12:26:12.2085088Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:12.2100953Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-12-04T12:26:12.2118638Z Entering 'third_party/benchmark'
2025-12-04T12:26:12.2129750Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:12.2139086Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:12.2148781Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-12-04T12:26:12.2163698Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:12.2174375Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-12-04T12:26:12.2187083Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:12.2200955Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-12-04T12:26:12.2209996Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:12.2221263Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-12-04T12:26:12.2237028Z Entering 'third_party/cutlass'
2025-12-04T12:26:12.2248144Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-12-04T12:26:12.2264775Z Entering 'third_party/fbgemm'
2025-12-04T12:26:12.2275996Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-12-04T12:26:12.2291531Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:12.2301172Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-12-04T12:26:12.2310829Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:12.2326998Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-12-04T12:26:12.2344819Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:12.2358255Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-12-04T12:26:12.2368108Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:12.2384189Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-12-04T12:26:12.2397168Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:12.2412908Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-12-04T12:26:12.2427358Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:12.2440511Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-12-04T12:26:12.2450828Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:12.2460746Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-12-04T12:26:12.2471828Z Entering 'third_party/flash-attention'
2025-12-04T12:26:12.2482035Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-12-04T12:26:12.2490401Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:12.2508521Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-12-04T12:26:12.2522256Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:12.2536202Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-12-04T12:26:12.2551039Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:12.2561321Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-12-04T12:26:12.2574891Z Entering 'third_party/fmt'
2025-12-04T12:26:12.2589932Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:12.2600196Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:12.2618718Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-12-04T12:26:12.2631078Z Entering 'third_party/gloo'
2025-12-04T12:26:12.2641322Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-12-04T12:26:12.2651022Z Entering 'third_party/googletest'
2025-12-04T12:26:12.2661207Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:12.2670599Z Entering 'third_party/ideep'
2025-12-04T12:26:12.2681951Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-12-04T12:26:12.2693763Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:12.2706124Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-12-04T12:26:12.2723888Z Entering 'third_party/ittapi'
2025-12-04T12:26:12.2736445Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-12-04T12:26:12.2751309Z Entering 'third_party/kineto'
2025-12-04T12:26:12.2770382Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-12-04T12:26:12.2780258Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:12.2795291Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-12-04T12:26:12.2804608Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:12.2816281Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-12-04T12:26:12.2825706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:12.2841525Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-12-04T12:26:12.2850582Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:12.2863774Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:12.2872281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:12.2886340Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-12-04T12:26:12.2895840Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:12.2908253Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-12-04T12:26:12.2924612Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:12.2937864Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-12-04T12:26:12.2947714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:12.2959447Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:12.2968361Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:12.2981335Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-12-04T12:26:12.2990784Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:12.3009616Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-12-04T12:26:12.3019595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:12.3036978Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T12:26:12.3047145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:12.3062613Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T12:26:12.3075938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:12.3086961Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T12:26:12.3102800Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:12.3120065Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:12.3129684Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:12.3141897Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:12.3157033Z Entering 'third_party/kleidiai'
2025-12-04T12:26:12.3172336Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-12-04T12:26:12.3183041Z Entering 'third_party/mimalloc'
2025-12-04T12:26:12.3193282Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-12-04T12:26:12.3203448Z Entering 'third_party/nlohmann'
2025-12-04T12:26:12.3218032Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-12-04T12:26:12.3229145Z Entering 'third_party/onnx'
2025-12-04T12:26:12.3239668Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-12-04T12:26:12.3253670Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:12.3270718Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:12.3283058Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:12.3301991Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-12-04T12:26:12.3312183Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:12.3324813Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:12.3334217Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:12.3348396Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:12.3358111Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:12.3371837Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-12-04T12:26:12.3380985Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:12.3396898Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-12-04T12:26:12.3407237Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:12.3421803Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-12-04T12:26:12.3437156Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:12.3448585Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-12-04T12:26:12.3464683Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:12.3474242Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T12:26:12.3482565Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:12.3497869Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T12:26:12.3508406Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:12.3524863Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T12:26:12.3536565Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:12.3553163Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-12-04T12:26:12.3571901Z Entering 'third_party/pocketfft'
2025-12-04T12:26:12.3583322Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-12-04T12:26:12.3591814Z Entering 'third_party/protobuf'
2025-12-04T12:26:12.3606473Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-12-04T12:26:12.3618072Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:12.3631887Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:12.3642626Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:12.3655001Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:12.3666385Z Entering 'third_party/psimd'
2025-12-04T12:26:12.3679228Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-12-04T12:26:12.3689524Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:12.3700562Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-12-04T12:26:12.3709955Z Entering 'third_party/pybind11'
2025-12-04T12:26:12.3724008Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:12.3734174Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:12.3745331Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-12-04T12:26:12.3754410Z Entering 'third_party/sleef'
2025-12-04T12:26:12.3769153Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-12-04T12:26:12.3778806Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:12.3792571Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-12-04T12:26:12.3802435Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:12.3814576Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:12.3826017Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:12.3837324Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-12-04T12:26:12.3852420Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:12.3867406Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-12-04T12:26:12.3877306Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:12.3891242Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:12.3900388Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:12.3913300Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-12-04T12:26:12.3940341Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.3960306Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.3979459Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.3993071Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4009062Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4025646Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4044049Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4058485Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4071560Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4090603Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4107472Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4123127Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4135993Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4151763Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4165565Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4181983Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4196022Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4209130Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4224158Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4238702Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4252617Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4266581Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4286836Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4304308Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4326346Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4340289Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4354634Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4369023Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4387191Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4401227Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4414133Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4435554Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4451712Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4465873Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4479951Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4494792Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4513718Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4528512Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4550142Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4571002Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4585472Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4602267Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4617032Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4631196Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4645640Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4662549Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4675798Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4690417Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4704474Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4719212Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4733651Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4748407Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4763167Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4781132Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4798919Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4814058Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4828719Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4849367Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4864344Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4879885Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4899555Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4913905Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4932401Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4949851Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4964844Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4980539Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.4996864Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5011210Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5026555Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5042598Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5058498Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5074375Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5088803Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5103811Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5117091Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5132213Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5146906Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5163333Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5178546Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5195073Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5208789Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T12:26:12.5226510Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-12-04T12:26:12.5248599Z ##[endgroup]
2025-12-04T12:26:12.5248788Z ##[group]Fetching the repository
2025-12-04T12:26:12.5252046Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-12-04T12:26:13.9504417Z [command]/usr/bin/git rev-parse --verify --quiet ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32^{object}
2025-12-04T12:26:13.9729547Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:13.9734665Z ##[endgroup]
2025-12-04T12:26:13.9735056Z ##[group]Determining the checkout info
2025-12-04T12:26:13.9737029Z ##[endgroup]
2025-12-04T12:26:13.9742367Z [command]/usr/bin/git sparse-checkout disable
2025-12-04T12:26:13.9832278Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-12-04T12:26:13.9855170Z ##[group]Checking out the ref
2025-12-04T12:26:13.9856427Z [command]/usr/bin/git checkout --progress --force ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:14.0169646Z HEAD is now at ffd9b0fb4355 Resolve collective autotuning test failure on arm (#168919)
2025-12-04T12:26:14.0175764Z ##[endgroup]
2025-12-04T12:26:14.0176220Z ##[group]Setting up auth for fetching submodules
2025-12-04T12:26:14.0181210Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-12-04T12:26:14.0208846Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-12-04T12:26:14.0225888Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-12-04T12:26:14.0247827Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-12-04T12:26:14.0262473Z ##[endgroup]
2025-12-04T12:26:14.0262691Z ##[group]Fetching submodules
2025-12-04T12:26:14.0264175Z [command]/usr/bin/git submodule sync --recursive
2025-12-04T12:26:14.0507948Z Synchronizing submodule url for 'android/libs/fbjni'
2025-12-04T12:26:14.0520921Z Synchronizing submodule url for 'third_party/FP16'
2025-12-04T12:26:14.0538326Z Synchronizing submodule url for 'third_party/FXdiv'
2025-12-04T12:26:14.0550141Z Synchronizing submodule url for 'third_party/NNPACK'
2025-12-04T12:26:14.0562367Z Synchronizing submodule url for 'third_party/NVTX'
2025-12-04T12:26:14.0577892Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:14.0591299Z Synchronizing submodule url for 'third_party/XNNPACK'
2025-12-04T12:26:14.0609582Z Synchronizing submodule url for 'third_party/aiter'
2025-12-04T12:26:14.0622412Z Synchronizing submodule url for 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:14.0638638Z Synchronizing submodule url for 'third_party/benchmark'
2025-12-04T12:26:14.0650159Z Synchronizing submodule url for 'third_party/composable_kernel'
2025-12-04T12:26:14.0666246Z Synchronizing submodule url for 'third_party/cpp-httplib'
2025-12-04T12:26:14.0680124Z Synchronizing submodule url for 'third_party/cpuinfo'
2025-12-04T12:26:14.0692525Z Synchronizing submodule url for 'third_party/cudnn_frontend'
2025-12-04T12:26:14.0705041Z Synchronizing submodule url for 'third_party/cutlass'
2025-12-04T12:26:14.0727195Z Synchronizing submodule url for 'third_party/fbgemm'
2025-12-04T12:26:14.0742320Z Synchronizing submodule url for 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:14.0753822Z Synchronizing submodule url for 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:14.0774886Z Synchronizing submodule url for 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:14.0790673Z Synchronizing submodule url for 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:14.0815784Z Synchronizing submodule url for 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:14.0830905Z Synchronizing submodule url for 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:14.0847932Z Synchronizing submodule url for 'third_party/fbgemm/external/json'
2025-12-04T12:26:14.0862912Z Synchronizing submodule url for 'third_party/flash-attention'
2025-12-04T12:26:14.0880960Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:14.0895640Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:14.0915947Z Synchronizing submodule url for 'third_party/flatbuffers'
2025-12-04T12:26:14.0930870Z Synchronizing submodule url for 'third_party/fmt'
2025-12-04T12:26:14.0946086Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:14.0958297Z Synchronizing submodule url for 'third_party/gloo'
2025-12-04T12:26:14.0971196Z Synchronizing submodule url for 'third_party/googletest'
2025-12-04T12:26:14.0989203Z Synchronizing submodule url for 'third_party/ideep'
2025-12-04T12:26:14.1002257Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:14.1021465Z Synchronizing submodule url for 'third_party/ittapi'
2025-12-04T12:26:14.1033023Z Synchronizing submodule url for 'third_party/kineto'
2025-12-04T12:26:14.1051467Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:14.1065377Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:14.1075263Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:14.1089647Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:14.1103703Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:14.1124623Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:14.1138919Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:14.1153035Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:14.1164961Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:14.1182895Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:14.1196030Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:14.1210050Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:14.1223323Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:14.1244027Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:14.1255370Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:14.1269549Z Synchronizing submodule url for 'third_party/kleidiai'
2025-12-04T12:26:14.1282478Z Synchronizing submodule url for 'third_party/mimalloc'
2025-12-04T12:26:14.1293854Z Synchronizing submodule url for 'third_party/nlohmann'
2025-12-04T12:26:14.1310711Z Synchronizing submodule url for 'third_party/onnx'
2025-12-04T12:26:14.1332910Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:14.1349729Z Synchronizing submodule url for 'third_party/opentelemetry-cpp'
2025-12-04T12:26:14.1364362Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:14.1375564Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:14.1390931Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:14.1401173Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:14.1412217Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:14.1423734Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:14.1440027Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:14.1455165Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:14.1469068Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:14.1479509Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:14.1500032Z Synchronizing submodule url for 'third_party/pocketfft'
2025-12-04T12:26:14.1513295Z Synchronizing submodule url for 'third_party/protobuf'
2025-12-04T12:26:14.1525499Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:14.1538221Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:14.1552903Z Synchronizing submodule url for 'third_party/psimd'
2025-12-04T12:26:14.1570865Z Synchronizing submodule url for 'third_party/pthreadpool'
2025-12-04T12:26:14.1582010Z Synchronizing submodule url for 'third_party/pybind11'
2025-12-04T12:26:14.1597931Z Synchronizing submodule url for 'third_party/python-peachpy'
2025-12-04T12:26:14.1617388Z Synchronizing submodule url for 'third_party/sleef'
2025-12-04T12:26:14.1628847Z Synchronizing submodule url for 'third_party/tensorpipe'
2025-12-04T12:26:14.1639073Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:14.1651258Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:14.1662951Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:14.1679563Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:14.1692704Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:14.1719175Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-12-04T12:26:14.1959124Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-12-04T12:26:14.2022336Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-12-04T12:26:14.2084406Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-12-04T12:26:14.2146435Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-12-04T12:26:14.2205678Z Submodule path 'third_party/NVTX': checked out '3ebbc93ded7285963bff932c678fa367eb393ba6'
2025-12-04T12:26:14.2263027Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1'
2025-12-04T12:26:14.2410295Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-12-04T12:26:14.2559454Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150'
2025-12-04T12:26:14.2729144Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf'
2025-12-04T12:26:14.2804236Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f'
2025-12-04T12:26:14.3007913Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977'
2025-12-04T12:26:14.3072718Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246'
2025-12-04T12:26:14.3135861Z Submodule path 'third_party/cpuinfo': checked out 'f858c30bcb16f8effd5ff46996f0514539e17abc'
2025-12-04T12:26:14.3209069Z Submodule path 'third_party/cudnn_frontend': checked out '0b1577c8c83401237d601d0d0db5210506705396'
2025-12-04T12:26:14.3324480Z Submodule path 'third_party/cutlass': checked out 'f88806b1e31dfa579842638740216dd41fc6c588'
2025-12-04T12:26:14.3458853Z Submodule path 'third_party/fbgemm': checked out 'c0b988d39a9e47c794d699f29930ed4d7c7e13a4'
2025-12-04T12:26:14.3521630Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea'
2025-12-04T12:26:14.3718727Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977'
2025-12-04T12:26:14.3780577Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349'
2025-12-04T12:26:14.3887847Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '98125ce499b0fdf7ffbe0e3052f5b8709f4840f8'
2025-12-04T12:26:14.3943447Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-12-04T12:26:14.3992691Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691'
2025-12-04T12:26:14.4078295Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03'
2025-12-04T12:26:14.4161996Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-12-04T12:26:14.4332565Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-12-04T12:26:14.4446017Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-12-04T12:26:14.4544523Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757'
2025-12-04T12:26:14.4605675Z Submodule path 'third_party/fmt': checked out '407c905e45ad75fc29bf0f9bb7c5c2fd3475976f'
2025-12-04T12:26:14.4669829Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-12-04T12:26:14.4730796Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341'
2025-12-04T12:26:14.4806626Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-12-04T12:26:14.4871112Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-12-04T12:26:14.5044360Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-12-04T12:26:14.5101582Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959'
2025-12-04T12:26:14.5191589Z Submodule path 'third_party/kineto': checked out '31f85df8fbd89c188f14ef10f1ec65379786b943'
2025-12-04T12:26:14.5282359Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1'
2025-12-04T12:26:14.5360374Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-12-04T12:26:14.5414522Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-12-04T12:26:14.5488169Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-12-04T12:26:14.5551849Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-12-04T12:26:14.5610628Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-12-04T12:26:14.5667696Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-12-04T12:26:14.5731221Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-12-04T12:26:14.5821287Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-12-04T12:26:14.5886530Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-12-04T12:26:14.5975961Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a'
2025-12-04T12:26:14.6060776Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159'
2025-12-04T12:26:14.6132070Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-12-04T12:26:14.6206079Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21'
2025-12-04T12:26:14.6278185Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-12-04T12:26:14.6369038Z Submodule path 'third_party/kleidiai': checked out 'd7770c89632329a9914ef1a90289917597639cbe'
2025-12-04T12:26:14.6468623Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e'
2025-12-04T12:26:14.6564342Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72'
2025-12-04T12:26:14.6719783Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83'
2025-12-04T12:26:14.6793225Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-12-04T12:26:14.6880881Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-12-04T12:26:14.6956073Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-12-04T12:26:14.7015553Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-12-04T12:26:14.7081052Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-12-04T12:26:14.7167130Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-12-04T12:26:14.7216414Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-12-04T12:26:14.7263635Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-12-04T12:26:14.7327912Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-12-04T12:26:14.7420295Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-12-04T12:26:14.7488796Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-12-04T12:26:14.7646292Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-12-04T12:26:14.7735762Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa'
2025-12-04T12:26:14.7890007Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-12-04T12:26:14.7964258Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-12-04T12:26:14.8033815Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-12-04T12:26:14.8084979Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-12-04T12:26:14.8135979Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-12-04T12:26:14.8205392Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8'
2025-12-04T12:26:14.8252675Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-12-04T12:26:14.8311728Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68'
2025-12-04T12:26:14.8369841Z Submodule path 'third_party/tensorpipe': checked out '2b4cd91092d335a697416b2a3cb398283246849d'
2025-12-04T12:26:14.8423989Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-12-04T12:26:14.8477950Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-12-04T12:26:14.8626708Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b'
2025-12-04T12:26:14.8693384Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-12-04T12:26:14.8757859Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-12-04T12:26:14.8783873Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-12-04T12:26:14.8974532Z Entering 'android/libs/fbjni'
2025-12-04T12:26:14.8993599Z Entering 'third_party/FP16'
2025-12-04T12:26:14.9015073Z Entering 'third_party/FXdiv'
2025-12-04T12:26:14.9039143Z Entering 'third_party/NNPACK'
2025-12-04T12:26:14.9061195Z Entering 'third_party/NVTX'
2025-12-04T12:26:14.9086502Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:14.9108422Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:14.9134546Z Entering 'third_party/aiter'
2025-12-04T12:26:14.9160041Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:14.9187375Z Entering 'third_party/benchmark'
2025-12-04T12:26:14.9209263Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:14.9236438Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:14.9255920Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:14.9282864Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:14.9308487Z Entering 'third_party/cutlass'
2025-12-04T12:26:14.9340242Z Entering 'third_party/fbgemm'
2025-12-04T12:26:14.9360667Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:14.9381228Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:14.9410687Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:14.9430945Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:14.9464978Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:14.9487577Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:14.9507964Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:14.9529560Z Entering 'third_party/flash-attention'
2025-12-04T12:26:14.9549505Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:14.9580713Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:14.9605817Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:14.9637132Z Entering 'third_party/fmt'
2025-12-04T12:26:14.9661396Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:14.9683231Z Entering 'third_party/gloo'
2025-12-04T12:26:14.9708135Z Entering 'third_party/googletest'
2025-12-04T12:26:14.9730465Z Entering 'third_party/ideep'
2025-12-04T12:26:14.9751266Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:14.9777593Z Entering 'third_party/ittapi'
2025-12-04T12:26:14.9799710Z Entering 'third_party/kineto'
2025-12-04T12:26:14.9822137Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:14.9839846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:14.9865344Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:14.9884880Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:14.9909205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:14.9931976Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:14.9963288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:14.9983697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:15.0001762Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:15.0023085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:15.0042521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:15.0062427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:15.0089696Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:15.0117251Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:15.0141305Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:15.0161294Z Entering 'third_party/kleidiai'
2025-12-04T12:26:15.0191500Z Entering 'third_party/mimalloc'
2025-12-04T12:26:15.0212562Z Entering 'third_party/nlohmann'
2025-12-04T12:26:15.0236062Z Entering 'third_party/onnx'
2025-12-04T12:26:15.0262789Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:15.0288128Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:15.0309394Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:15.0330137Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:15.0351529Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:15.0377435Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:15.0398031Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:15.0416448Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:15.0435807Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:15.0453554Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:15.0480826Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:15.0502966Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:15.0531080Z Entering 'third_party/pocketfft'
2025-12-04T12:26:15.0554073Z Entering 'third_party/protobuf'
2025-12-04T12:26:15.0574866Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:15.0598404Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:15.0623573Z Entering 'third_party/psimd'
2025-12-04T12:26:15.0646630Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:15.0667576Z Entering 'third_party/pybind11'
2025-12-04T12:26:15.0687248Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:15.0708144Z Entering 'third_party/sleef'
2025-12-04T12:26:15.0730664Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:15.0750250Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:15.0778635Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:15.0810873Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:15.0832474Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:15.0857403Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:15.0896715Z ##[endgroup]
2025-12-04T12:26:15.0896945Z ##[group]Persisting credentials for submodules
2025-12-04T12:26:15.0904259Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-12-04T12:26:15.1095918Z Entering 'android/libs/fbjni'
2025-12-04T12:26:15.1112278Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1112522Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1130263Z Entering 'third_party/FP16'
2025-12-04T12:26:15.1148720Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1148914Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1171259Z Entering 'third_party/FXdiv'
2025-12-04T12:26:15.1192602Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1192787Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1214799Z Entering 'third_party/NNPACK'
2025-12-04T12:26:15.1232565Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1232731Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1252138Z Entering 'third_party/NVTX'
2025-12-04T12:26:15.1273633Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1273783Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1292281Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:15.1313841Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1313976Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1339267Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:15.1360215Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1360349Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1392340Z Entering 'third_party/aiter'
2025-12-04T12:26:15.1411833Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1411954Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1436080Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:15.1451156Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1451285Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1473951Z Entering 'third_party/benchmark'
2025-12-04T12:26:15.1492889Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1493015Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1510559Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:15.1530287Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1530605Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1553275Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:15.1574442Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1574573Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1597476Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:15.1617635Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1617763Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1638439Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:15.1652545Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1652680Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1672694Z Entering 'third_party/cutlass'
2025-12-04T12:26:15.1691312Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1691452Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1715370Z Entering 'third_party/fbgemm'
2025-12-04T12:26:15.1731001Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1731137Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1751166Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:15.1767480Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1767613Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1794679Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:15.1813454Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1813773Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1843866Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:15.1859287Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1859568Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1883077Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:15.1901650Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1901900Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1925849Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:15.1941283Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1941521Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1959973Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:15.1976409Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1976640Z url.https://github.com/.insteadof
2025-12-04T12:26:15.1993715Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:15.2008390Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2008570Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2028704Z Entering 'third_party/flash-attention'
2025-12-04T12:26:15.2044754Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2044945Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2063090Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:15.2080900Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2081218Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2111677Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:15.2127738Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2128016Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2155549Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:15.2171278Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2171440Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2194014Z Entering 'third_party/fmt'
2025-12-04T12:26:15.2209108Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2209308Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2236642Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:15.2250681Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2250869Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2269423Z Entering 'third_party/gloo'
2025-12-04T12:26:15.2284581Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2284768Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2305610Z Entering 'third_party/googletest'
2025-12-04T12:26:15.2319767Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2319924Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2344007Z Entering 'third_party/ideep'
2025-12-04T12:26:15.2358221Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2358485Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2374717Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:15.2394009Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2394143Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2421476Z Entering 'third_party/ittapi'
2025-12-04T12:26:15.2439712Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2439831Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2461535Z Entering 'third_party/kineto'
2025-12-04T12:26:15.2476714Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2476959Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2498357Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:15.2514646Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2514775Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2532159Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:15.2546013Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2546153Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2571167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:15.2586972Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2587110Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2606826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:15.2624357Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2624481Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2643729Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:15.2658246Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2658363Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2680872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:15.2694362Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2694483Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2717013Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:15.2731997Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2732119Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2754447Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:15.2770884Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2770999Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2787905Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:15.2809285Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2809406Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2831475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:15.2845767Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2845888Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2863769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:15.2877881Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2878002Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2896556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:15.2910420Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2910543Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2934187Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:15.2954291Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2954410Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2977311Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:15.2996691Z url.https://github.com/.insteadof
2025-12-04T12:26:15.2996877Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3014923Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:15.3029777Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3029910Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3048478Z Entering 'third_party/kleidiai'
2025-12-04T12:26:15.3066675Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3066833Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3086032Z Entering 'third_party/mimalloc'
2025-12-04T12:26:15.3099754Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3099879Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3120914Z Entering 'third_party/nlohmann'
2025-12-04T12:26:15.3134134Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3134251Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3152802Z Entering 'third_party/onnx'
2025-12-04T12:26:15.3166847Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3167323Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3194825Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:15.3215829Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3216113Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3242783Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:15.3258605Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3258877Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3278534Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:15.3295253Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3295573Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3312579Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:15.3328691Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3328907Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3350942Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:15.3364538Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3364881Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3382494Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:15.3397385Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3397593Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3414116Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:15.3430569Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3430765Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3453112Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:15.3467031Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3467163Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3489781Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:15.3506822Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3506946Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3526818Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:15.3547680Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3547805Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3567826Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:15.3580370Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3580543Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3598780Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:15.3612455Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3612736Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3645064Z Entering 'third_party/pocketfft'
2025-12-04T12:26:15.3660007Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3660168Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3676314Z Entering 'third_party/protobuf'
2025-12-04T12:26:15.3688893Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3689034Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3708558Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:15.3722247Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3722412Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3741369Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:15.3755595Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3755729Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3778752Z Entering 'third_party/psimd'
2025-12-04T12:26:15.3792348Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3793154Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3811074Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:15.3828928Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3829089Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3847921Z Entering 'third_party/pybind11'
2025-12-04T12:26:15.3860503Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3860630Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3877033Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:15.3889373Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3889506Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3906642Z Entering 'third_party/sleef'
2025-12-04T12:26:15.3919280Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3919430Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3936222Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:15.3950819Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3950942Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3970984Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:15.3984094Z url.https://github.com/.insteadof
2025-12-04T12:26:15.3984221Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4001739Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:15.4017441Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4017567Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4035287Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:15.4048125Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4048244Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4065445Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:15.4078976Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4079103Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4094056Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:15.4109107Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4109235Z url.https://github.com/.insteadof
2025-12-04T12:26:15.4146367Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-12-04T12:26:15.4301047Z Entering 'android/libs/fbjni'
2025-12-04T12:26:15.4328628Z file:/home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-12-04T12:26:15.4340205Z Entering 'third_party/FP16'
2025-12-04T12:26:15.4366353Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-12-04T12:26:15.4378153Z Entering 'third_party/FXdiv'
2025-12-04T12:26:15.4404514Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-12-04T12:26:15.4413853Z Entering 'third_party/NNPACK'
2025-12-04T12:26:15.4434943Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-12-04T12:26:15.4443673Z Entering 'third_party/NVTX'
2025-12-04T12:26:15.4464222Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-12-04T12:26:15.4474026Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:15.4493534Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-12-04T12:26:15.4502512Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:15.4529154Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-12-04T12:26:15.4545345Z Entering 'third_party/aiter'
2025-12-04T12:26:15.4564989Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-12-04T12:26:15.4574993Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:15.4601466Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-12-04T12:26:15.4619927Z Entering 'third_party/benchmark'
2025-12-04T12:26:15.4647054Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:15.4661222Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:15.4687186Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-12-04T12:26:15.4701268Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:15.4724173Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-12-04T12:26:15.4734048Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:15.4754163Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-12-04T12:26:15.4764549Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:15.4787334Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-12-04T12:26:15.4804514Z Entering 'third_party/cutlass'
2025-12-04T12:26:15.4827037Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-12-04T12:26:15.4842377Z Entering 'third_party/fbgemm'
2025-12-04T12:26:15.4865042Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-12-04T12:26:15.4883895Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:15.4905559Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-12-04T12:26:15.4915384Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:15.4938948Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-12-04T12:26:15.4952606Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:15.4980443Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-12-04T12:26:15.4991876Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:15.5017437Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-12-04T12:26:15.5029665Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:15.5055820Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-12-04T12:26:15.5066258Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:15.5092518Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-12-04T12:26:15.5102313Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:15.5123449Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-12-04T12:26:15.5139508Z Entering 'third_party/flash-attention'
2025-12-04T12:26:15.5166624Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-12-04T12:26:15.5177820Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:15.5199181Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-12-04T12:26:15.5212110Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:15.5232103Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-12-04T12:26:15.5247050Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:15.5268754Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-12-04T12:26:15.5283043Z Entering 'third_party/fmt'
2025-12-04T12:26:15.5307528Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:15.5318159Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:15.5342518Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-12-04T12:26:15.5352891Z Entering 'third_party/gloo'
2025-12-04T12:26:15.5375419Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-12-04T12:26:15.5385960Z Entering 'third_party/googletest'
2025-12-04T12:26:15.5408871Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:15.5419272Z Entering 'third_party/ideep'
2025-12-04T12:26:15.5442880Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-12-04T12:26:15.5453387Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:15.5480097Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-12-04T12:26:15.5495696Z Entering 'third_party/ittapi'
2025-12-04T12:26:15.5519374Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-12-04T12:26:15.5529732Z Entering 'third_party/kineto'
2025-12-04T12:26:15.5552531Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-12-04T12:26:15.5563221Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:15.5586116Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-12-04T12:26:15.5600876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:15.5625531Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-12-04T12:26:15.5636998Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:15.5665691Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-12-04T12:26:15.5677868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:15.5699577Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:15.5712502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:15.5748805Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-12-04T12:26:15.5763714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:15.5789599Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-12-04T12:26:15.5806541Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:15.5826888Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-12-04T12:26:15.5838024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:15.5863490Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:15.5881105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:15.5908968Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-12-04T12:26:15.5919801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:15.5945542Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-12-04T12:26:15.5955288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:15.5976617Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T12:26:15.5991266Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:15.6020919Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T12:26:15.6032832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:15.6057994Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T12:26:15.6079937Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:15.6103632Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-12-04T12:26:15.6116349Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:15.6143308Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:15.6162824Z Entering 'third_party/kleidiai'
2025-12-04T12:26:15.6185525Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-12-04T12:26:15.6196730Z Entering 'third_party/mimalloc'
2025-12-04T12:26:15.6217950Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-12-04T12:26:15.6235014Z Entering 'third_party/nlohmann'
2025-12-04T12:26:15.6268416Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-12-04T12:26:15.6278863Z Entering 'third_party/onnx'
2025-12-04T12:26:15.6311482Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-12-04T12:26:15.6334312Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:15.6366300Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:15.6379805Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:15.6400447Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-12-04T12:26:15.6411812Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:15.6438877Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:15.6453790Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:15.6477795Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:15.6488332Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:15.6511544Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-12-04T12:26:15.6521636Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:15.6549050Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-12-04T12:26:15.6559809Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:15.6582508Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-12-04T12:26:15.6592023Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:15.6617737Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-12-04T12:26:15.6626683Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:15.6652427Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T12:26:15.6672124Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:15.6695371Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T12:26:15.6704840Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:15.6729296Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T12:26:15.6741471Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:15.6772275Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-12-04T12:26:15.6791541Z Entering 'third_party/pocketfft'
2025-12-04T12:26:15.6817866Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-12-04T12:26:15.6828342Z Entering 'third_party/protobuf'
2025-12-04T12:26:15.6849501Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-12-04T12:26:15.6860869Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:15.6883679Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T12:26:15.6895154Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:15.6921619Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:15.6933602Z Entering 'third_party/psimd'
2025-12-04T12:26:15.6956873Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-12-04T12:26:15.6967337Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:15.6994195Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-12-04T12:26:15.7007114Z Entering 'third_party/pybind11'
2025-12-04T12:26:15.7029223Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:15.7041199Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:15.7067412Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-12-04T12:26:15.7077871Z Entering 'third_party/sleef'
2025-12-04T12:26:15.7106499Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-12-04T12:26:15.7116471Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:15.7141139Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-12-04T12:26:15.7151069Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:15.7176688Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-12-04T12:26:15.7186336Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:15.7208633Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-12-04T12:26:15.7219080Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:15.7242006Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-12-04T12:26:15.7253800Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:15.7274210Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T12:26:15.7282936Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:15.7306059Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-12-04T12:26:15.7499154Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-12-04T12:26:15.7687703Z Entering 'android/libs/fbjni'
2025-12-04T12:26:15.7709178Z Entering 'third_party/FP16'
2025-12-04T12:26:15.7728099Z Entering 'third_party/FXdiv'
2025-12-04T12:26:15.7752900Z Entering 'third_party/NNPACK'
2025-12-04T12:26:15.7777228Z Entering 'third_party/NVTX'
2025-12-04T12:26:15.7801366Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:15.7828661Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:15.7855921Z Entering 'third_party/aiter'
2025-12-04T12:26:15.7878729Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:15.7909450Z Entering 'third_party/benchmark'
2025-12-04T12:26:15.7931758Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:15.7957694Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:15.7991379Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:15.8013968Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:15.8037633Z Entering 'third_party/cutlass'
2025-12-04T12:26:15.8072852Z Entering 'third_party/fbgemm'
2025-12-04T12:26:15.8100104Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:15.8124254Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:15.8153252Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:15.8173859Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:15.8198643Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:15.8221879Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:15.8240997Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:15.8270808Z Entering 'third_party/flash-attention'
2025-12-04T12:26:15.8294007Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:15.8317096Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:15.8342255Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:15.8364549Z Entering 'third_party/fmt'
2025-12-04T12:26:15.8383935Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:15.8404281Z Entering 'third_party/gloo'
2025-12-04T12:26:15.8423979Z Entering 'third_party/googletest'
2025-12-04T12:26:15.8444299Z Entering 'third_party/ideep'
2025-12-04T12:26:15.8462052Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:15.8490177Z Entering 'third_party/ittapi'
2025-12-04T12:26:15.8509493Z Entering 'third_party/kineto'
2025-12-04T12:26:15.8528867Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:15.8551984Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:15.8574204Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:15.8594033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:15.8613367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:15.8644090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:15.8671619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:15.8691826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:15.8711216Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:15.8729558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:15.8750029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:15.8768679Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:15.8790844Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:15.8814163Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:15.8832788Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:15.8852075Z Entering 'third_party/kleidiai'
2025-12-04T12:26:15.8871785Z Entering 'third_party/mimalloc'
2025-12-04T12:26:15.8892074Z Entering 'third_party/nlohmann'
2025-12-04T12:26:15.8911929Z Entering 'third_party/onnx'
2025-12-04T12:26:15.8938603Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:15.8960469Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:15.8980182Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:15.8998189Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:15.9024214Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:15.9045573Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:15.9064951Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:15.9084071Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:15.9105074Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:15.9122279Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:15.9141140Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:15.9166271Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:15.9192907Z Entering 'third_party/pocketfft'
2025-12-04T12:26:15.9216566Z Entering 'third_party/protobuf'
2025-12-04T12:26:15.9244127Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:15.9266646Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:15.9294660Z Entering 'third_party/psimd'
2025-12-04T12:26:15.9314318Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:15.9336428Z Entering 'third_party/pybind11'
2025-12-04T12:26:15.9360408Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:15.9379092Z Entering 'third_party/sleef'
2025-12-04T12:26:15.9404764Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:15.9424177Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:15.9446137Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:15.9467791Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:15.9487917Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:15.9507395Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:15.9542232Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-12-04T12:26:15.9736953Z Entering 'android/libs/fbjni'
2025-12-04T12:26:15.9769787Z Entering 'third_party/FP16'
2025-12-04T12:26:15.9797114Z Entering 'third_party/FXdiv'
2025-12-04T12:26:15.9828273Z Entering 'third_party/NNPACK'
2025-12-04T12:26:15.9852927Z Entering 'third_party/NVTX'
2025-12-04T12:26:15.9875827Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T12:26:15.9898370Z Entering 'third_party/XNNPACK'
2025-12-04T12:26:15.9925404Z Entering 'third_party/aiter'
2025-12-04T12:26:15.9953896Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T12:26:15.9990302Z Entering 'third_party/benchmark'
2025-12-04T12:26:16.0017364Z Entering 'third_party/composable_kernel'
2025-12-04T12:26:16.0048138Z Entering 'third_party/cpp-httplib'
2025-12-04T12:26:16.0068453Z Entering 'third_party/cpuinfo'
2025-12-04T12:26:16.0091601Z Entering 'third_party/cudnn_frontend'
2025-12-04T12:26:16.0118778Z Entering 'third_party/cutlass'
2025-12-04T12:26:16.0143617Z Entering 'third_party/fbgemm'
2025-12-04T12:26:16.0173319Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T12:26:16.0199525Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T12:26:16.0223788Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T12:26:16.0251472Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T12:26:16.0277923Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T12:26:16.0300541Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T12:26:16.0321693Z Entering 'third_party/fbgemm/external/json'
2025-12-04T12:26:16.0344264Z Entering 'third_party/flash-attention'
2025-12-04T12:26:16.0364698Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T12:26:16.0389780Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T12:26:16.0423567Z Entering 'third_party/flatbuffers'
2025-12-04T12:26:16.0456735Z Entering 'third_party/fmt'
2025-12-04T12:26:16.0478318Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T12:26:16.0502866Z Entering 'third_party/gloo'
2025-12-04T12:26:16.0527942Z Entering 'third_party/googletest'
2025-12-04T12:26:16.0555637Z Entering 'third_party/ideep'
2025-12-04T12:26:16.0587861Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T12:26:16.0616366Z Entering 'third_party/ittapi'
2025-12-04T12:26:16.0647288Z Entering 'third_party/kineto'
2025-12-04T12:26:16.0671873Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T12:26:16.0698061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T12:26:16.0722580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T12:26:16.0742411Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T12:26:16.0767659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T12:26:16.0796307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T12:26:16.0827090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T12:26:16.0854263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T12:26:16.0874917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T12:26:16.0894709Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T12:26:16.0918207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T12:26:16.0943409Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:16.0974804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:16.1009277Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T12:26:16.1037641Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T12:26:16.1069875Z Entering 'third_party/kleidiai'
2025-12-04T12:26:16.1096034Z Entering 'third_party/mimalloc'
2025-12-04T12:26:16.1124797Z Entering 'third_party/nlohmann'
2025-12-04T12:26:16.1151257Z Entering 'third_party/onnx'
2025-12-04T12:26:16.1182418Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T12:26:16.1218593Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T12:26:16.1243374Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T12:26:16.1263743Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T12:26:16.1292888Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T12:26:16.1323398Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T12:26:16.1348904Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T12:26:16.1372095Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T12:26:16.1391870Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T12:26:16.1413763Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T12:26:16.1437784Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T12:26:16.1468524Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T12:26:16.1503886Z Entering 'third_party/pocketfft'
2025-12-04T12:26:16.1529423Z Entering 'third_party/protobuf'
2025-12-04T12:26:16.1555149Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T12:26:16.1577005Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T12:26:16.1610648Z Entering 'third_party/psimd'
2025-12-04T12:26:16.1632199Z Entering 'third_party/pthreadpool'
2025-12-04T12:26:16.1654443Z Entering 'third_party/pybind11'
2025-12-04T12:26:16.1674306Z Entering 'third_party/python-peachpy'
2025-12-04T12:26:16.1703180Z Entering 'third_party/sleef'
2025-12-04T12:26:16.1730654Z Entering 'third_party/tensorpipe'
2025-12-04T12:26:16.1755908Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T12:26:16.1774335Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T12:26:16.1797333Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T12:26:16.1821728Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T12:26:16.1838830Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T12:26:16.1873834Z ##[endgroup]
2025-12-04T12:26:16.2050871Z [command]/usr/bin/git log -1 --format=%H
2025-12-04T12:26:16.2154273Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:16.2270680Z Prepare all required actions
2025-12-04T12:26:16.2270944Z Getting action download info
2025-12-04T12:26:16.5074425Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076)
2025-12-04T12:26:17.3188262Z ##[group]Run ./.github/actions/setup-rocm
2025-12-04T12:26:17.3188413Z env:
2025-12-04T12:26:17.3188501Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.3188600Z ##[endgroup]
2025-12-04T12:26:17.3201248Z ##[group]Run dpkg -l | grep -E "  rocm"
2025-12-04T12:26:17.3201392Z dpkg -l | grep -E "  rocm"
2025-12-04T12:26:17.3205807Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.3205954Z env:
2025-12-04T12:26:17.3206042Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.3206150Z ##[endgroup]
2025-12-04T12:26:17.3269140Z ii  rocm-cmake                   0.14.0.60401-83~22.04                   amd64        rocm-cmake built using CMake
2025-12-04T12:26:17.3269383Z ii  rocm-core                    6.4.1.60401-83~22.04                    amd64        ROCm Runtime software stack
2025-12-04T12:26:17.3269606Z ii  rocm-dbgapi                  0.77.2.60401-83~22.04                   amd64        Library to provide AMD GPU debugger API
2025-12-04T12:26:17.3269865Z ii  rocm-debug-agent             2.0.4.60401-83~22.04                    amd64        Radeon Open Compute Debug Agent (ROCdebug-agent)
2025-12-04T12:26:17.3270122Z ii  rocm-dev                     6.4.1.60401-83~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-12-04T12:26:17.3270361Z ii  rocm-device-libs             1.0.0.60401-83~22.04                    amd64        Radeon Open Compute - device libraries
2025-12-04T12:26:17.3270576Z ii  rocm-gdb                     15.2.60401-83~22.04                     amd64        ROCgdb
2025-12-04T12:26:17.3270775Z ii  rocm-llvm                    19.0.0.25184.60401-83~22.04             amd64        ROCm core compiler
2025-12-04T12:26:17.3270989Z ii  rocm-opencl                  2.0.0.60401-83~22.04                    amd64        clr built using CMake
2025-12-04T12:26:17.3271205Z ii  rocm-opencl-dev              2.0.0.60401-83~22.04                    amd64        clr built using CMake
2025-12-04T12:26:17.3271424Z ii  rocm-smi-lib                 7.5.0.60401-83~22.04                    amd64        AMD System Management libraries
2025-12-04T12:26:17.3272005Z ii  rocm-utils                   6.4.1.60401-83~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-12-04T12:26:17.3272250Z ii  rocminfo                     1.0.0.60401-83~22.04                    amd64        Radeon Open Compute (ROCm) Runtime rocminfo tool
2025-12-04T12:26:17.3285576Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-12-04T12:26:17.3285846Z # ignore expansion of "docker ps -q" since it could be empty
2025-12-04T12:26:17.3286004Z # shellcheck disable=SC2046
2025-12-04T12:26:17.3286146Z docker stop $(docker ps -q) || true
2025-12-04T12:26:17.3286276Z # Prune all stopped containers.
2025-12-04T12:26:17.3286401Z docker container prune -f
2025-12-04T12:26:17.3290720Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.3290865Z env:
2025-12-04T12:26:17.3290953Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.3291065Z ##[endgroup]
2025-12-04T12:26:17.3521026Z docker: 'docker stop' requires at least 1 argument
2025-12-04T12:26:17.3521269Z 
2025-12-04T12:26:17.3521392Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2025-12-04T12:26:17.3521557Z 
2025-12-04T12:26:17.3521660Z See 'docker stop --help' for more information
2025-12-04T12:26:17.3623925Z Total reclaimed space: 0B
2025-12-04T12:26:17.3652648Z ##[group]Run cat /etc/os-release || true
2025-12-04T12:26:17.3652841Z cat /etc/os-release || true
2025-12-04T12:26:17.3653006Z cat /etc/apt/sources.list.d/rocm.list || true
2025-12-04T12:26:17.3653336Z cat /opt/rocm/.info/version || true
2025-12-04T12:26:17.3653477Z whoami
2025-12-04T12:26:17.3658125Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.3658299Z env:
2025-12-04T12:26:17.3658402Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.3658524Z ##[endgroup]
2025-12-04T12:26:17.3681843Z PRETTY_NAME="Ubuntu 22.04.5 LTS"
2025-12-04T12:26:17.3681987Z NAME="Ubuntu"
2025-12-04T12:26:17.3682080Z VERSION_ID="22.04"
2025-12-04T12:26:17.3682181Z VERSION="22.04.5 LTS (Jammy Jellyfish)"
2025-12-04T12:26:17.3682303Z VERSION_CODENAME=jammy
2025-12-04T12:26:17.3682400Z ID=ubuntu
2025-12-04T12:26:17.3682482Z ID_LIKE=debian
2025-12-04T12:26:17.3682605Z HOME_URL="https://www.ubuntu.com/"
2025-12-04T12:26:17.3682734Z SUPPORT_URL="https://help.ubuntu.com/"
2025-12-04T12:26:17.3682907Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2025-12-04T12:26:17.3683116Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2025-12-04T12:26:17.3683307Z UBUNTU_CODENAME=jammy
2025-12-04T12:26:17.3687787Z deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.4.1 jammy main
2025-12-04T12:26:17.3692011Z 6.4.1-83
2025-12-04T12:26:17.3698153Z runner
2025-12-04T12:26:17.3709675Z ##[group]Run dpkg -l | grep -E "  amdgpu"
2025-12-04T12:26:17.3709827Z dpkg -l | grep -E "  amdgpu"
2025-12-04T12:26:17.3713041Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.3713178Z env:
2025-12-04T12:26:17.3713262Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.3713360Z ##[endgroup]
2025-12-04T12:26:17.3767984Z ii  amdgpu-core                  1:6.4.60401-2164967.22.04               all          Core meta package for unified amdgpu driver.
2025-12-04T12:26:17.3768242Z ii  amdgpu-install               6.4.60401-2164967.22.04                 all          AMDGPU driver repository and installer
2025-12-04T12:26:17.3790199Z ##[group]Run rocm-smi
2025-12-04T12:26:17.3790366Z rocm-smi
2025-12-04T12:26:17.3795381Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.3795577Z env:
2025-12-04T12:26:17.3795691Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.3795825Z ##[endgroup]
2025-12-04T12:26:17.4425406Z 
2025-12-04T12:26:17.4425418Z 
2025-12-04T12:26:17.4425774Z ============================================ ROCm System Management Interface ============================================
2025-12-04T12:26:17.4426571Z ====================================================== Concise Info ======================================================
2025-12-04T12:26:17.4427228Z Device  Node  IDs              Temp        Power     Partitions          SCLK  MCLK    Fan  Perf    PwrCap   VRAM%  GPU%  
2025-12-04T12:26:17.4428299Z               (DID,     GUID)  (Junction)  (Socket)  (Mem, Compute, ID)                                                   
2025-12-04T12:26:17.4428814Z ==========================================================================================================================
2025-12-04T12:26:17.4429650Z 0       3     0x74a5,   51110  27.0°C      119.0W    NPS1, SPX, 0        N/A   900Mhz  0%   manual  1000.0W  0%     0%    
2025-12-04T12:26:17.4429944Z 1       5     0x74a5,   2987   28.0°C      113.0W    NPS1, SPX, 0        N/A   900Mhz  0%   manual  1000.0W  0%     0%    
2025-12-04T12:26:17.4430215Z 2       4     0x74a5,   61326  25.0°C      112.0W    NPS1, SPX, 0        N/A   900Mhz  0%   manual  1000.0W  0%     0%    
2025-12-04T12:26:17.4430502Z 3       2     0x74a5,   9091   28.0°C      113.0W    NPS1, SPX, 0        N/A   900Mhz  0%   manual  1000.0W  0%     0%    
2025-12-04T12:26:17.4430694Z ==========================================================================================================================
2025-12-04T12:26:17.4430864Z ================================================== End of ROCm SMI Log ===================================================
2025-12-04T12:26:17.4491883Z ##[group]Run rocminfo
2025-12-04T12:26:17.4492053Z rocminfo
2025-12-04T12:26:17.4497439Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.4497648Z env:
2025-12-04T12:26:17.4497767Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.4497913Z ##[endgroup]
2025-12-04T12:26:17.5397934Z ROCk module version 6.12.12 is loaded
2025-12-04T12:26:17.5398103Z =====================    
2025-12-04T12:26:17.5398250Z HSA System Attributes    
2025-12-04T12:26:17.5398365Z =====================    
2025-12-04T12:26:17.5398507Z Runtime Version:         1.15
2025-12-04T12:26:17.5398644Z Runtime Ext Version:     1.7
2025-12-04T12:26:17.5398836Z System Timestamp Freq.:  1000.000000MHz
2025-12-04T12:26:17.5399054Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-12-04T12:26:17.5399285Z Machine Model:           LARGE                              
2025-12-04T12:26:17.5399479Z System Endianness:       LITTLE                             
2025-12-04T12:26:17.5399647Z Mwaitx:                  DISABLED
2025-12-04T12:26:17.5399784Z XNACK enabled:           NO
2025-12-04T12:26:17.5399913Z DMAbuf Support:          YES
2025-12-04T12:26:17.5414258Z VMM Support:             YES
2025-12-04T12:26:17.5414335Z 
2025-12-04T12:26:17.5414401Z ==========               
2025-12-04T12:26:17.5414503Z HSA Agents               
2025-12-04T12:26:17.5414601Z ==========               
2025-12-04T12:26:17.5414720Z *******                  
2025-12-04T12:26:17.5414814Z Agent 1                  
2025-12-04T12:26:17.5414911Z *******                  
2025-12-04T12:26:17.5415027Z   Name:                    AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:26:17.5415218Z   Uuid:                    CPU-XX                             
2025-12-04T12:26:17.5415379Z   Marketing Name:          AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:26:17.5415601Z   Vendor Name:             CPU                                
2025-12-04T12:26:17.5415793Z   Feature:                 None specified                     
2025-12-04T12:26:17.5415953Z   Profile:                 FULL_PROFILE                       
2025-12-04T12:26:17.5416127Z   Float Round Mode:        NEAR                               
2025-12-04T12:26:17.5416302Z   Max Queue Number:        0(0x0)                             
2025-12-04T12:26:17.5416464Z   Queue Min Size:          0(0x0)                             
2025-12-04T12:26:17.5416626Z   Queue Max Size:          0(0x0)                             
2025-12-04T12:26:17.5416999Z   Queue Type:              MULTI                              
2025-12-04T12:26:17.5417136Z   Node:                    0                                  
2025-12-04T12:26:17.5417279Z   Device Type:             CPU                                
2025-12-04T12:26:17.5417411Z   Cache Info:              
2025-12-04T12:26:17.5417573Z     L1:                      49152(0xc000) KB                   
2025-12-04T12:26:17.5417734Z   Chip ID:                 0(0x0)                             
2025-12-04T12:26:17.5417886Z   ASIC Revision:           0(0x0)                             
2025-12-04T12:26:17.5418067Z   Cacheline Size:          64(0x40)                           
2025-12-04T12:26:17.5418225Z   Max Clock Freq. (MHz):   3300                               
2025-12-04T12:26:17.5418391Z   BDFID:                   0                                  
2025-12-04T12:26:17.5418541Z   Internal Node ID:        0                                  
2025-12-04T12:26:17.5418693Z   Compute Unit:            64                                 
2025-12-04T12:26:17.5418841Z   SIMDs per CU:            0                                  
2025-12-04T12:26:17.5418995Z   Shader Engines:          0                                  
2025-12-04T12:26:17.5419149Z   Shader Arrs. per Eng.:   0                                  
2025-12-04T12:26:17.5419322Z   WatchPts on Addr. Ranges:1                                  
2025-12-04T12:26:17.5419466Z   Memory Properties:       
2025-12-04T12:26:17.5419595Z   Features:                None
2025-12-04T12:26:17.5419744Z   Pool Info:               
2025-12-04T12:26:17.5419905Z     Pool 1                   
2025-12-04T12:26:17.5420084Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:26:17.5420241Z       Size:                    1584734448(0x5e7520f0) KB          
2025-12-04T12:26:17.5420389Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5420546Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5420719Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:26:17.5420913Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5421074Z       Accessible by all:       TRUE                               
2025-12-04T12:26:17.5421240Z     Pool 2                   
2025-12-04T12:26:17.5421373Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:26:17.5421553Z       Size:                    1584734448(0x5e7520f0) KB          
2025-12-04T12:26:17.5421700Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5421869Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5422039Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:26:17.5422199Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5422358Z       Accessible by all:       TRUE                               
2025-12-04T12:26:17.5422497Z     Pool 3                   
2025-12-04T12:26:17.5422641Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-12-04T12:26:17.5422795Z       Size:                    1584734448(0x5e7520f0) KB          
2025-12-04T12:26:17.5422945Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5423133Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5423310Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:26:17.5423471Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5423642Z       Accessible by all:       TRUE                               
2025-12-04T12:26:17.5423785Z     Pool 4                   
2025-12-04T12:26:17.5423912Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:26:17.5424061Z       Size:                    1584734448(0x5e7520f0) KB          
2025-12-04T12:26:17.5424242Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5424419Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5424606Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:26:17.5424764Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5424921Z       Accessible by all:       TRUE                               
2025-12-04T12:26:17.5425059Z   ISA Info:                
2025-12-04T12:26:17.5425165Z *******                  
2025-12-04T12:26:17.5425269Z Agent 2                  
2025-12-04T12:26:17.5425395Z *******                  
2025-12-04T12:26:17.5425529Z   Name:                    AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:26:17.5425677Z   Uuid:                    CPU-XX                             
2025-12-04T12:26:17.5425845Z   Marketing Name:          AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:26:17.5425999Z   Vendor Name:             CPU                                
2025-12-04T12:26:17.5426150Z   Feature:                 None specified                     
2025-12-04T12:26:17.5426308Z   Profile:                 FULL_PROFILE                       
2025-12-04T12:26:17.5426487Z   Float Round Mode:        NEAR                               
2025-12-04T12:26:17.5426638Z   Max Queue Number:        0(0x0)                             
2025-12-04T12:26:17.5426835Z   Queue Min Size:          0(0x0)                             
2025-12-04T12:26:17.5426987Z   Queue Max Size:          0(0x0)                             
2025-12-04T12:26:17.5427189Z   Queue Type:              MULTI                              
2025-12-04T12:26:17.5427335Z   Node:                    1                                  
2025-12-04T12:26:17.5427479Z   Device Type:             CPU                                
2025-12-04T12:26:17.5427611Z   Cache Info:              
2025-12-04T12:26:17.5427728Z     L1:                      49152(0xc000) KB                   
2025-12-04T12:26:17.5427876Z   Chip ID:                 0(0x0)                             
2025-12-04T12:26:17.5428052Z   ASIC Revision:           0(0x0)                             
2025-12-04T12:26:17.5428239Z   Cacheline Size:          64(0x40)                           
2025-12-04T12:26:17.5428389Z   Max Clock Freq. (MHz):   3300                               
2025-12-04T12:26:17.5428533Z   BDFID:                   0                                  
2025-12-04T12:26:17.5428678Z   Internal Node ID:        1                                  
2025-12-04T12:26:17.5428827Z   Compute Unit:            64                                 
2025-12-04T12:26:17.5428978Z   SIMDs per CU:            0                                  
2025-12-04T12:26:17.5429151Z   Shader Engines:          0                                  
2025-12-04T12:26:17.5429335Z   Shader Arrs. per Eng.:   0                                  
2025-12-04T12:26:17.5429514Z   WatchPts on Addr. Ranges:1                                  
2025-12-04T12:26:17.5429656Z   Memory Properties:       
2025-12-04T12:26:17.5429776Z   Features:                None
2025-12-04T12:26:17.5429918Z   Pool Info:               
2025-12-04T12:26:17.5430018Z     Pool 1                   
2025-12-04T12:26:17.5430184Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:26:17.5430347Z       Size:                    1585355632(0x5e7e9b70) KB          
2025-12-04T12:26:17.5430502Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5430660Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5430839Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:26:17.5431005Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5431175Z       Accessible by all:       TRUE                               
2025-12-04T12:26:17.5431311Z     Pool 2                   
2025-12-04T12:26:17.5431441Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:26:17.5431627Z       Size:                    1585355632(0x5e7e9b70) KB          
2025-12-04T12:26:17.5431776Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5431931Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5432105Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:26:17.5432275Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5432434Z       Accessible by all:       TRUE                               
2025-12-04T12:26:17.5432574Z     Pool 3                   
2025-12-04T12:26:17.5432734Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-12-04T12:26:17.5432886Z       Size:                    1585355632(0x5e7e9b70) KB          
2025-12-04T12:26:17.5433027Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5433183Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5433399Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:26:17.5433564Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5433723Z       Accessible by all:       TRUE                               
2025-12-04T12:26:17.5433856Z     Pool 4                   
2025-12-04T12:26:17.5433983Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:26:17.5434142Z       Size:                    1585355632(0x5e7e9b70) KB          
2025-12-04T12:26:17.5434333Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5434485Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5434636Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:26:17.5434803Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5434960Z       Accessible by all:       TRUE                               
2025-12-04T12:26:17.5435090Z   ISA Info:                
2025-12-04T12:26:17.5435220Z *******                  
2025-12-04T12:26:17.5435335Z Agent 3                  
2025-12-04T12:26:17.5435425Z *******                  
2025-12-04T12:26:17.5435539Z   Name:                    gfx942                             
2025-12-04T12:26:17.5435678Z   Uuid:                    GPU-dd105c7c944d0f95               
2025-12-04T12:26:17.5435828Z   Marketing Name:          AMD Instinct MI325X                
2025-12-04T12:26:17.5436023Z   Vendor Name:             AMD                                
2025-12-04T12:26:17.5436167Z   Feature:                 KERNEL_DISPATCH                    
2025-12-04T12:26:17.5436410Z   Profile:                 BASE_PROFILE                       
2025-12-04T12:26:17.5436559Z   Float Round Mode:        NEAR                               
2025-12-04T12:26:17.5436719Z   Max Queue Number:        128(0x80)                          
2025-12-04T12:26:17.5436903Z   Queue Min Size:          64(0x40)                           
2025-12-04T12:26:17.5437060Z   Queue Max Size:          131072(0x20000)                    
2025-12-04T12:26:17.5437201Z   Queue Type:              MULTI                              
2025-12-04T12:26:17.5437336Z   Node:                    2                                  
2025-12-04T12:26:17.5437469Z   Device Type:             GPU                                
2025-12-04T12:26:17.5437596Z   Cache Info:              
2025-12-04T12:26:17.5437727Z     L1:                      32(0x20) KB                        
2025-12-04T12:26:17.5437852Z     L2:                      4096(0x1000) KB                    
2025-12-04T12:26:17.5437992Z     L3:                      262144(0x40000) KB                 
2025-12-04T12:26:17.5438121Z   Chip ID:                 29861(0x74a5)                      
2025-12-04T12:26:17.5438261Z   ASIC Revision:           1(0x1)                             
2025-12-04T12:26:17.5438412Z   Cacheline Size:          128(0x80)                          
2025-12-04T12:26:17.5438612Z   Max Clock Freq. (MHz):   2100                               
2025-12-04T12:26:17.5438752Z   BDFID:                   29952                              
2025-12-04T12:26:17.5438893Z   Internal Node ID:        2                                  
2025-12-04T12:26:17.5439035Z   Compute Unit:            304                                
2025-12-04T12:26:17.5439232Z   SIMDs per CU:            4                                  
2025-12-04T12:26:17.5439401Z   Shader Engines:          32                                 
2025-12-04T12:26:17.5439555Z   Shader Arrs. per Eng.:   1                                  
2025-12-04T12:26:17.5439757Z   WatchPts on Addr. Ranges:4                                  
2025-12-04T12:26:17.5439950Z   Coherent Host Access:    FALSE                              
2025-12-04T12:26:17.5440105Z   Memory Properties:       
2025-12-04T12:26:17.5440242Z   Features:                KERNEL_DISPATCH 
2025-12-04T12:26:17.5440389Z   Fast F16 Operation:      TRUE                               
2025-12-04T12:26:17.5440541Z   Wavefront Size:          64(0x40)                           
2025-12-04T12:26:17.5440705Z   Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5440844Z   Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5440965Z     x                        1024(0x400)                        
2025-12-04T12:26:17.5441087Z     y                        1024(0x400)                        
2025-12-04T12:26:17.5441233Z     z                        1024(0x400)                        
2025-12-04T12:26:17.5441367Z   Max Waves Per CU:        32(0x20)                           
2025-12-04T12:26:17.5441561Z   Max Work-item Per CU:    2048(0x800)                        
2025-12-04T12:26:17.5441710Z   Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5441841Z   Grid Max Size per Dimension:
2025-12-04T12:26:17.5441953Z     x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5442088Z     y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5442212Z     z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5442361Z   Max fbarriers/Workgrp:   32                                 
2025-12-04T12:26:17.5448262Z   Packet Processor uCode:: 185                                
2025-12-04T12:26:17.5448437Z   SDMA engine uCode::      24                                 
2025-12-04T12:26:17.5448595Z   IOMMU Support::          None                               
2025-12-04T12:26:17.5448728Z   Pool Info:               
2025-12-04T12:26:17.5448828Z     Pool 1                   
2025-12-04T12:26:17.5448964Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:26:17.5449113Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5449263Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5449421Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5449587Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5449752Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5449912Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5450047Z     Pool 2                   
2025-12-04T12:26:17.5450174Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:26:17.5450320Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5450460Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5450614Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5450768Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5450924Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5451078Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5451283Z     Pool 3                   
2025-12-04T12:26:17.5451406Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:26:17.5451547Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5451686Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5451837Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5451993Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5452157Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5452319Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5452452Z     Pool 4                   
2025-12-04T12:26:17.5452574Z       Segment:                 GROUP                              
2025-12-04T12:26:17.5452715Z       Size:                    64(0x40) KB                        
2025-12-04T12:26:17.5452856Z       Allocatable:             FALSE                              
2025-12-04T12:26:17.5453017Z       Alloc Granule:           0KB                                
2025-12-04T12:26:17.5453183Z       Alloc Recommended Granule:0KB                                
2025-12-04T12:26:17.5453346Z       Alloc Alignment:         0KB                                
2025-12-04T12:26:17.5453509Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5453645Z   ISA Info:                
2025-12-04T12:26:17.5453755Z     ISA 1                    
2025-12-04T12:26:17.5453929Z       Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
2025-12-04T12:26:17.5454097Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:26:17.5454255Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:26:17.5454410Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5454563Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5454719Z       Fast f16:                TRUE                               
2025-12-04T12:26:17.5454864Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5455007Z       Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5455137Z         x                        1024(0x400)                        
2025-12-04T12:26:17.5455269Z         y                        1024(0x400)                        
2025-12-04T12:26:17.5455399Z         z                        1024(0x400)                        
2025-12-04T12:26:17.5455552Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5455691Z       Grid Max Size per Dimension:
2025-12-04T12:26:17.5455817Z         x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5455949Z         y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5456084Z         z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5456384Z       FBarrier Max Size:       32                                 
2025-12-04T12:26:17.5456521Z     ISA 2                    
2025-12-04T12:26:17.5456668Z       Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
2025-12-04T12:26:17.5456889Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:26:17.5457051Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:26:17.5457217Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5457384Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5457538Z       Fast f16:                TRUE                               
2025-12-04T12:26:17.5457694Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5457837Z       Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5457969Z         x                        1024(0x400)                        
2025-12-04T12:26:17.5458150Z         y                        1024(0x400)                        
2025-12-04T12:26:17.5458278Z         z                        1024(0x400)                        
2025-12-04T12:26:17.5458425Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5458568Z       Grid Max Size per Dimension:
2025-12-04T12:26:17.5458685Z         x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5458821Z         y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5458948Z         z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5459098Z       FBarrier Max Size:       32                                 
2025-12-04T12:26:17.5459239Z *******                  
2025-12-04T12:26:17.5459338Z Agent 4                  
2025-12-04T12:26:17.5459440Z *******                  
2025-12-04T12:26:17.5459557Z   Name:                    gfx942                             
2025-12-04T12:26:17.5459708Z   Uuid:                    GPU-6f0efc92194cd6bf               
2025-12-04T12:26:17.5459868Z   Marketing Name:          AMD Instinct MI325X                
2025-12-04T12:26:17.5460023Z   Vendor Name:             AMD                                
2025-12-04T12:26:17.5460235Z   Feature:                 KERNEL_DISPATCH                    
2025-12-04T12:26:17.5460391Z   Profile:                 BASE_PROFILE                       
2025-12-04T12:26:17.5460544Z   Float Round Mode:        NEAR                               
2025-12-04T12:26:17.5460703Z   Max Queue Number:        128(0x80)                          
2025-12-04T12:26:17.5460904Z   Queue Min Size:          64(0x40)                           
2025-12-04T12:26:17.5461054Z   Queue Max Size:          131072(0x20000)                    
2025-12-04T12:26:17.5461210Z   Queue Type:              MULTI                              
2025-12-04T12:26:17.5461352Z   Node:                    3                                  
2025-12-04T12:26:17.5461500Z   Device Type:             GPU                                
2025-12-04T12:26:17.5461643Z   Cache Info:              
2025-12-04T12:26:17.5461759Z     L1:                      32(0x20) KB                        
2025-12-04T12:26:17.5461896Z     L2:                      4096(0x1000) KB                    
2025-12-04T12:26:17.5462032Z     L3:                      262144(0x40000) KB                 
2025-12-04T12:26:17.5462167Z   Chip ID:                 29861(0x74a5)                      
2025-12-04T12:26:17.5462319Z   ASIC Revision:           1(0x1)                             
2025-12-04T12:26:17.5462484Z   Cacheline Size:          128(0x80)                          
2025-12-04T12:26:17.5462637Z   Max Clock Freq. (MHz):   2100                               
2025-12-04T12:26:17.5462790Z   BDFID:                   1280                               
2025-12-04T12:26:17.5462936Z   Internal Node ID:        3                                  
2025-12-04T12:26:17.5463092Z   Compute Unit:            304                                
2025-12-04T12:26:17.5463249Z   SIMDs per CU:            4                                  
2025-12-04T12:26:17.5463400Z   Shader Engines:          32                                 
2025-12-04T12:26:17.5463559Z   Shader Arrs. per Eng.:   1                                  
2025-12-04T12:26:17.5463724Z   WatchPts on Addr. Ranges:4                                  
2025-12-04T12:26:17.5463883Z   Coherent Host Access:    FALSE                              
2025-12-04T12:26:17.5464030Z   Memory Properties:       
2025-12-04T12:26:17.5464146Z   Features:                KERNEL_DISPATCH 
2025-12-04T12:26:17.5464296Z   Fast F16 Operation:      TRUE                               
2025-12-04T12:26:17.5464457Z   Wavefront Size:          64(0x40)                           
2025-12-04T12:26:17.5464612Z   Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5464761Z   Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5464891Z     x                        1024(0x400)                        
2025-12-04T12:26:17.5465047Z     y                        1024(0x400)                        
2025-12-04T12:26:17.5465179Z     z                        1024(0x400)                        
2025-12-04T12:26:17.5465320Z   Max Waves Per CU:        32(0x20)                           
2025-12-04T12:26:17.5465481Z   Max Work-item Per CU:    2048(0x800)                        
2025-12-04T12:26:17.5465641Z   Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5465777Z   Grid Max Size per Dimension:
2025-12-04T12:26:17.5465896Z     x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5466033Z     y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5466161Z     z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5466311Z   Max fbarriers/Workgrp:   32                                 
2025-12-04T12:26:17.5466477Z   Packet Processor uCode:: 185                                
2025-12-04T12:26:17.5466648Z   SDMA engine uCode::      24                                 
2025-12-04T12:26:17.5466849Z   IOMMU Support::          None                               
2025-12-04T12:26:17.5466984Z   Pool Info:               
2025-12-04T12:26:17.5467093Z     Pool 1                   
2025-12-04T12:26:17.5467227Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:26:17.5467376Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5467532Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5467693Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5467892Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5468062Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5468222Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5468366Z     Pool 2                   
2025-12-04T12:26:17.5468509Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:26:17.5468657Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5468809Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5468969Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5469130Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5469296Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5469453Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5469597Z     Pool 3                   
2025-12-04T12:26:17.5469728Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:26:17.5469873Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5470024Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5470185Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5470346Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5470514Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5470676Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5470812Z     Pool 4                   
2025-12-04T12:26:17.5470940Z       Segment:                 GROUP                              
2025-12-04T12:26:17.5471081Z       Size:                    64(0x40) KB                        
2025-12-04T12:26:17.5471234Z       Allocatable:             FALSE                              
2025-12-04T12:26:17.5471393Z       Alloc Granule:           0KB                                
2025-12-04T12:26:17.5471554Z       Alloc Recommended Granule:0KB                                
2025-12-04T12:26:17.5471719Z       Alloc Alignment:         0KB                                
2025-12-04T12:26:17.5471918Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5472050Z   ISA Info:                
2025-12-04T12:26:17.5472148Z     ISA 1                    
2025-12-04T12:26:17.5472270Z       Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
2025-12-04T12:26:17.5472429Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:26:17.5472581Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:26:17.5472735Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5472893Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5473040Z       Fast f16:                TRUE                               
2025-12-04T12:26:17.5473199Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5473343Z       Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5473475Z         x                        1024(0x400)                        
2025-12-04T12:26:17.5473611Z         y                        1024(0x400)                        
2025-12-04T12:26:17.5473744Z         z                        1024(0x400)                        
2025-12-04T12:26:17.5473891Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5474031Z       Grid Max Size per Dimension:
2025-12-04T12:26:17.5474156Z         x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5474292Z         y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5474450Z         z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5474600Z       FBarrier Max Size:       32                                 
2025-12-04T12:26:17.5474736Z     ISA 2                    
2025-12-04T12:26:17.5474880Z       Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
2025-12-04T12:26:17.5475056Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:26:17.5475219Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:26:17.5475384Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5475553Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5475706Z       Fast f16:                TRUE                               
2025-12-04T12:26:17.5475864Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5476015Z       Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5476142Z         x                        1024(0x400)                        
2025-12-04T12:26:17.5476278Z         y                        1024(0x400)                        
2025-12-04T12:26:17.5476404Z         z                        1024(0x400)                        
2025-12-04T12:26:17.5476548Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5476688Z       Grid Max Size per Dimension:
2025-12-04T12:26:17.5476846Z         x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5476981Z         y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5477114Z         z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5477258Z       FBarrier Max Size:       32                                 
2025-12-04T12:26:17.5477396Z *******                  
2025-12-04T12:26:17.5477495Z Agent 5                  
2025-12-04T12:26:17.5477599Z *******                  
2025-12-04T12:26:17.5477719Z   Name:                    gfx942                             
2025-12-04T12:26:17.5477865Z   Uuid:                    GPU-f619d38b8c3dfd9b               
2025-12-04T12:26:17.5478024Z   Marketing Name:          AMD Instinct MI325X                
2025-12-04T12:26:17.5478185Z   Vendor Name:             AMD                                
2025-12-04T12:26:17.5478337Z   Feature:                 KERNEL_DISPATCH                    
2025-12-04T12:26:17.5478494Z   Profile:                 BASE_PROFILE                       
2025-12-04T12:26:17.5478687Z   Float Round Mode:        NEAR                               
2025-12-04T12:26:17.5478846Z   Max Queue Number:        128(0x80)                          
2025-12-04T12:26:17.5479003Z   Queue Min Size:          64(0x40)                           
2025-12-04T12:26:17.5479150Z   Queue Max Size:          131072(0x20000)                    
2025-12-04T12:26:17.5479355Z   Queue Type:              MULTI                              
2025-12-04T12:26:17.5479502Z   Node:                    4                                  
2025-12-04T12:26:17.5479644Z   Device Type:             GPU                                
2025-12-04T12:26:17.5479780Z   Cache Info:              
2025-12-04T12:26:17.5479893Z     L1:                      32(0x20) KB                        
2025-12-04T12:26:17.5480029Z     L2:                      4096(0x1000) KB                    
2025-12-04T12:26:17.5480160Z     L3:                      262144(0x40000) KB                 
2025-12-04T12:26:17.5480300Z   Chip ID:                 29861(0x74a5)                      
2025-12-04T12:26:17.5480447Z   ASIC Revision:           1(0x1)                             
2025-12-04T12:26:17.5480603Z   Cacheline Size:          128(0x80)                          
2025-12-04T12:26:17.5480756Z   Max Clock Freq. (MHz):   2100                               
2025-12-04T12:26:17.5480907Z   BDFID:                   25856                              
2025-12-04T12:26:17.5481059Z   Internal Node ID:        4                                  
2025-12-04T12:26:17.5481247Z   Compute Unit:            304                                
2025-12-04T12:26:17.5481402Z   SIMDs per CU:            4                                  
2025-12-04T12:26:17.5481553Z   Shader Engines:          32                                 
2025-12-04T12:26:17.5481714Z   Shader Arrs. per Eng.:   1                                  
2025-12-04T12:26:17.5481878Z   WatchPts on Addr. Ranges:4                                  
2025-12-04T12:26:17.5482038Z   Coherent Host Access:    FALSE                              
2025-12-04T12:26:17.5482182Z   Memory Properties:       
2025-12-04T12:26:17.5482302Z   Features:                KERNEL_DISPATCH 
2025-12-04T12:26:17.5482439Z   Fast F16 Operation:      TRUE                               
2025-12-04T12:26:17.5482594Z   Wavefront Size:          64(0x40)                           
2025-12-04T12:26:17.5482745Z   Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5482889Z   Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5483015Z     x                        1024(0x400)                        
2025-12-04T12:26:17.5483139Z     y                        1024(0x400)                        
2025-12-04T12:26:17.5483263Z     z                        1024(0x400)                        
2025-12-04T12:26:17.5483401Z   Max Waves Per CU:        32(0x20)                           
2025-12-04T12:26:17.5483552Z   Max Work-item Per CU:    2048(0x800)                        
2025-12-04T12:26:17.5483711Z   Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5483842Z   Grid Max Size per Dimension:
2025-12-04T12:26:17.5483957Z     x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5484085Z     y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5484211Z     z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5484357Z   Max fbarriers/Workgrp:   32                                 
2025-12-04T12:26:17.5484518Z   Packet Processor uCode:: 185                                
2025-12-04T12:26:17.5484675Z   SDMA engine uCode::      24                                 
2025-12-04T12:26:17.5484828Z   IOMMU Support::          None                               
2025-12-04T12:26:17.5484964Z   Pool Info:               
2025-12-04T12:26:17.5485064Z     Pool 1                   
2025-12-04T12:26:17.5485194Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:26:17.5485371Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5485519Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5485675Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5485834Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5485997Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5486155Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5486288Z     Pool 2                   
2025-12-04T12:26:17.5486420Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:26:17.5486565Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5486711Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5486915Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5487078Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5487240Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5487398Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5487530Z     Pool 3                   
2025-12-04T12:26:17.5487655Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:26:17.5487796Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5487941Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5488133Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5488291Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5488452Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5488608Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5488744Z     Pool 4                   
2025-12-04T12:26:17.5488866Z       Segment:                 GROUP                              
2025-12-04T12:26:17.5489007Z       Size:                    64(0x40) KB                        
2025-12-04T12:26:17.5489146Z       Allocatable:             FALSE                              
2025-12-04T12:26:17.5489300Z       Alloc Granule:           0KB                                
2025-12-04T12:26:17.5489457Z       Alloc Recommended Granule:0KB                                
2025-12-04T12:26:17.5489620Z       Alloc Alignment:         0KB                                
2025-12-04T12:26:17.5489783Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5489970Z   ISA Info:                
2025-12-04T12:26:17.5490074Z     ISA 1                    
2025-12-04T12:26:17.5490204Z       Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
2025-12-04T12:26:17.5490364Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:26:17.5490528Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:26:17.5490681Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5490845Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5490996Z       Fast f16:                TRUE                               
2025-12-04T12:26:17.5491140Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5491279Z       Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5491404Z         x                        1024(0x400)                        
2025-12-04T12:26:17.5491527Z         y                        1024(0x400)                        
2025-12-04T12:26:17.5491649Z         z                        1024(0x400)                        
2025-12-04T12:26:17.5491782Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5491919Z       Grid Max Size per Dimension:
2025-12-04T12:26:17.5492083Z         x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5492211Z         y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5492341Z         z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5492484Z       FBarrier Max Size:       32                                 
2025-12-04T12:26:17.5492613Z     ISA 2                    
2025-12-04T12:26:17.5492749Z       Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
2025-12-04T12:26:17.5492918Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:26:17.5493250Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:26:17.5493487Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5493681Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5493834Z       Fast f16:                TRUE                               
2025-12-04T12:26:17.5493996Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5494137Z       Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5494265Z         x                        1024(0x400)                        
2025-12-04T12:26:17.5494392Z         y                        1024(0x400)                        
2025-12-04T12:26:17.5494514Z         z                        1024(0x400)                        
2025-12-04T12:26:17.5494664Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5494798Z       Grid Max Size per Dimension:
2025-12-04T12:26:17.5494956Z         x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5495090Z         y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5495216Z         z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5495359Z       FBarrier Max Size:       32                                 
2025-12-04T12:26:17.5495498Z *******                  
2025-12-04T12:26:17.5495595Z Agent 6                  
2025-12-04T12:26:17.5495695Z *******                  
2025-12-04T12:26:17.5495804Z   Name:                    gfx942                             
2025-12-04T12:26:17.5495972Z   Uuid:                    GPU-8052c781c18bf05c               
2025-12-04T12:26:17.5496128Z   Marketing Name:          AMD Instinct MI325X                
2025-12-04T12:26:17.5496284Z   Vendor Name:             AMD                                
2025-12-04T12:26:17.5496441Z   Feature:                 KERNEL_DISPATCH                    
2025-12-04T12:26:17.5496600Z   Profile:                 BASE_PROFILE                       
2025-12-04T12:26:17.5496809Z   Float Round Mode:        NEAR                               
2025-12-04T12:26:17.5496971Z   Max Queue Number:        128(0x80)                          
2025-12-04T12:26:17.5497122Z   Queue Min Size:          64(0x40)                           
2025-12-04T12:26:17.5497274Z   Queue Max Size:          131072(0x20000)                    
2025-12-04T12:26:17.5497433Z   Queue Type:              MULTI                              
2025-12-04T12:26:17.5497574Z   Node:                    5                                  
2025-12-04T12:26:17.5497721Z   Device Type:             GPU                                
2025-12-04T12:26:17.5497860Z   Cache Info:              
2025-12-04T12:26:17.5497975Z     L1:                      32(0x20) KB                        
2025-12-04T12:26:17.5498114Z     L2:                      4096(0x1000) KB                    
2025-12-04T12:26:17.5498246Z     L3:                      262144(0x40000) KB                 
2025-12-04T12:26:17.5498386Z   Chip ID:                 29861(0x74a5)                      
2025-12-04T12:26:17.5498538Z   ASIC Revision:           1(0x1)                             
2025-12-04T12:26:17.5498690Z   Cacheline Size:          128(0x80)                          
2025-12-04T12:26:17.5498849Z   Max Clock Freq. (MHz):   2100                               
2025-12-04T12:26:17.5499051Z   BDFID:                   5376                               
2025-12-04T12:26:17.5499198Z   Internal Node ID:        5                                  
2025-12-04T12:26:17.5499355Z   Compute Unit:            304                                
2025-12-04T12:26:17.5499510Z   SIMDs per CU:            4                                  
2025-12-04T12:26:17.5499664Z   Shader Engines:          32                                 
2025-12-04T12:26:17.5499824Z   Shader Arrs. per Eng.:   1                                  
2025-12-04T12:26:17.5499984Z   WatchPts on Addr. Ranges:4                                  
2025-12-04T12:26:17.5500149Z   Coherent Host Access:    FALSE                              
2025-12-04T12:26:17.5500295Z   Memory Properties:       
2025-12-04T12:26:17.5500409Z   Features:                KERNEL_DISPATCH 
2025-12-04T12:26:17.5500554Z   Fast F16 Operation:      TRUE                               
2025-12-04T12:26:17.5500713Z   Wavefront Size:          64(0x40)                           
2025-12-04T12:26:17.5500874Z   Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5501020Z   Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5501145Z     x                        1024(0x400)                        
2025-12-04T12:26:17.5501281Z     y                        1024(0x400)                        
2025-12-04T12:26:17.5501416Z     z                        1024(0x400)                        
2025-12-04T12:26:17.5501554Z   Max Waves Per CU:        32(0x20)                           
2025-12-04T12:26:17.5501749Z   Max Work-item Per CU:    2048(0x800)                        
2025-12-04T12:26:17.5501907Z   Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5502042Z   Grid Max Size per Dimension:
2025-12-04T12:26:17.5502159Z     x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5502284Z     y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5502419Z     z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5502565Z   Max fbarriers/Workgrp:   32                                 
2025-12-04T12:26:17.5502726Z   Packet Processor uCode:: 185                                
2025-12-04T12:26:17.5502888Z   SDMA engine uCode::      24                                 
2025-12-04T12:26:17.5503046Z   IOMMU Support::          None                               
2025-12-04T12:26:17.5503179Z   Pool Info:               
2025-12-04T12:26:17.5503284Z     Pool 1                   
2025-12-04T12:26:17.5503413Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:26:17.5503571Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5503723Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5503878Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5504046Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5504215Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5504372Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5504508Z     Pool 2                   
2025-12-04T12:26:17.5504640Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:26:17.5504787Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5504936Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5505089Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5505254Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5505418Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5505573Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5505709Z     Pool 3                   
2025-12-04T12:26:17.5505868Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:26:17.5506012Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:26:17.5506161Z       Allocatable:             TRUE                               
2025-12-04T12:26:17.5506315Z       Alloc Granule:           4KB                                
2025-12-04T12:26:17.5506481Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:26:17.5506649Z       Alloc Alignment:         4KB                                
2025-12-04T12:26:17.5506847Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5506990Z     Pool 4                   
2025-12-04T12:26:17.5507118Z       Segment:                 GROUP                              
2025-12-04T12:26:17.5507258Z       Size:                    64(0x40) KB                        
2025-12-04T12:26:17.5507407Z       Allocatable:             FALSE                              
2025-12-04T12:26:17.5507570Z       Alloc Granule:           0KB                                
2025-12-04T12:26:17.5507730Z       Alloc Recommended Granule:0KB                                
2025-12-04T12:26:17.5507895Z       Alloc Alignment:         0KB                                
2025-12-04T12:26:17.5508051Z       Accessible by all:       FALSE                              
2025-12-04T12:26:17.5508192Z   ISA Info:                
2025-12-04T12:26:17.5508301Z     ISA 1                    
2025-12-04T12:26:17.5508431Z       Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
2025-12-04T12:26:17.5508647Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:26:17.5508816Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:26:17.5508975Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5509145Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5509297Z       Fast f16:                TRUE                               
2025-12-04T12:26:17.5509460Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5509611Z       Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5509739Z         x                        1024(0x400)                        
2025-12-04T12:26:17.5509872Z         y                        1024(0x400)                        
2025-12-04T12:26:17.5510007Z         z                        1024(0x400)                        
2025-12-04T12:26:17.5510147Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5510289Z       Grid Max Size per Dimension:
2025-12-04T12:26:17.5510413Z         x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5510551Z         y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5510686Z         z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5510831Z       FBarrier Max Size:       32                                 
2025-12-04T12:26:17.5510978Z     ISA 2                    
2025-12-04T12:26:17.5511122Z       Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
2025-12-04T12:26:17.5511296Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:26:17.5511462Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:26:17.5511625Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5511805Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:26:17.5511966Z       Fast f16:                TRUE                               
2025-12-04T12:26:17.5512120Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:26:17.5512271Z       Workgroup Max Size per Dimension:
2025-12-04T12:26:17.5512402Z         x                        1024(0x400)                        
2025-12-04T12:26:17.5512529Z         y                        1024(0x400)                        
2025-12-04T12:26:17.5512700Z         z                        1024(0x400)                        
2025-12-04T12:26:17.5512845Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:26:17.5512982Z       Grid Max Size per Dimension:
2025-12-04T12:26:17.5513108Z         x                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5513243Z         y                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5513375Z         z                        4294967295(0xffffffff)             
2025-12-04T12:26:17.5513523Z       FBarrier Max Size:       32                                 
2025-12-04T12:26:17.5513659Z *** Done ***             
2025-12-04T12:26:17.5523797Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-12-04T12:26:17.5523984Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-12-04T12:26:17.5524264Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-12-04T12:26:17.5524537Z if [[ $ngpu -eq 0 ]]; then
2025-12-04T12:26:17.5524692Z     echo "Error: Failed to detect any GPUs on the runner"
2025-12-04T12:26:17.5524836Z     echo "$msg"
2025-12-04T12:26:17.5524945Z     exit 1
2025-12-04T12:26:17.5525041Z fi
2025-12-04T12:26:17.5528235Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.5528385Z env:
2025-12-04T12:26:17.5528474Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.5528580Z ##[endgroup]
2025-12-04T12:26:17.6492375Z ##[group]Run pytorch/pytorch/.github/actions/diskspace-cleanup@main
2025-12-04T12:26:17.6492563Z with:
2025-12-04T12:26:17.6492658Z   diskspace-cutoff: 70
2025-12-04T12:26:17.6492762Z env:
2025-12-04T12:26:17.6492855Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.6492960Z ##[endgroup]
2025-12-04T12:26:17.6528969Z ##[group]Run set -ex
2025-12-04T12:26:17.6529111Z set -ex
2025-12-04T12:26:17.6529211Z diskspace_cutoff=70
2025-12-04T12:26:17.6529369Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-12-04T12:26:17.6529532Z if [ ! -d "$docker_root_dir" ]; then
2025-12-04T12:26:17.6529731Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-12-04T12:26:17.6529919Z     exit 0
2025-12-04T12:26:17.6530012Z fi
2025-12-04T12:26:17.6530172Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-12-04T12:26:17.6530504Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-12-04T12:26:17.6530788Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-12-04T12:26:17.6530933Z     docker system prune -af
2025-12-04T12:26:17.6531123Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-12-04T12:26:17.6531341Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-12-04T12:26:17.6531500Z         diskspace_cutoff_int=$((diskspace_cutoff + 0))
2025-12-04T12:26:17.6531656Z         difference=$((100 - diskspace_cutoff_int))
2025-12-04T12:26:17.6531865Z         echo "Error: Available diskspace is less than $difference percent. Not enough diskspace."
2025-12-04T12:26:17.6532052Z         echo "$msg"
2025-12-04T12:26:17.6532156Z         exit 1
2025-12-04T12:26:17.6532250Z     else
2025-12-04T12:26:17.6532359Z         difference=$((diskspace - diskspace_new))
2025-12-04T12:26:17.6532512Z         echo "Diskspace saved: $difference percent"
2025-12-04T12:26:17.6532640Z     fi
2025-12-04T12:26:17.6532730Z fi
2025-12-04T12:26:17.6536969Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.6537110Z env:
2025-12-04T12:26:17.6537323Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.6537424Z ##[endgroup]
2025-12-04T12:26:17.6555574Z + diskspace_cutoff=70
2025-12-04T12:26:17.6559503Z ++ docker info -f '{{.DockerRootDir}}'
2025-12-04T12:26:17.6883941Z + docker_root_dir=/home/runner/docker-data
2025-12-04T12:26:17.6884375Z + '[' '!' -d /home/runner/docker-data ']'
2025-12-04T12:26:17.6889746Z ++ df -H --output=pcent /home/runner/docker-data
2025-12-04T12:26:17.6890179Z ++ sed -n 2p
2025-12-04T12:26:17.6893379Z ++ sed s/%//
2025-12-04T12:26:17.6893731Z ++ sed 's/ //'
2025-12-04T12:26:17.6909727Z + diskspace=' 3'
2025-12-04T12:26:17.6910095Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-12-04T12:26:17.6910414Z + [[  3 -ge 70 ]]
2025-12-04T12:26:17.6939848Z ##[group]Run RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-12-04T12:26:17.6940118Z RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-12-04T12:26:17.6940289Z rm -rf "${RUNNER_ARTIFACT_DIR}"
2025-12-04T12:26:17.6940454Z mkdir -p "${RUNNER_ARTIFACT_DIR}"
2025-12-04T12:26:17.6940649Z echo "RUNNER_ARTIFACT_DIR=${RUNNER_ARTIFACT_DIR}" >> "${GITHUB_ENV}"
2025-12-04T12:26:17.6940823Z 
2025-12-04T12:26:17.6940960Z RUNNER_TEST_RESULTS_DIR="${RUNNER_TEMP}/test-results"
2025-12-04T12:26:17.6941135Z rm -rf "${RUNNER_TEST_RESULTS_DIR}"
2025-12-04T12:26:17.6941280Z mkdir -p "${RUNNER_TEST_RESULTS_DIR}"
2025-12-04T12:26:17.6941478Z echo "RUNNER_TEST_RESULTS_DIR=${RUNNER_TEST_RESULTS_DIR}" >> "${GITHUB_ENV}"
2025-12-04T12:26:17.6941664Z 
2025-12-04T12:26:17.6941966Z RUNNER_DOCS_DIR="${RUNNER_TEMP}/docs"
2025-12-04T12:26:17.6942114Z rm -rf "${RUNNER_DOCS_DIR}"
2025-12-04T12:26:17.6942247Z mkdir -p "${RUNNER_DOCS_DIR}"
2025-12-04T12:26:17.6942416Z echo "RUNNER_DOCS_DIR=${RUNNER_DOCS_DIR}" >> "${GITHUB_ENV}"
2025-12-04T12:26:17.6947091Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.6947259Z env:
2025-12-04T12:26:17.6947362Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.6947475Z ##[endgroup]
2025-12-04T12:26:17.7026431Z ##[group]Run env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:26:17.7026677Z env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:26:17.7026940Z env | grep '^CI' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:26:17.7031126Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.7031285Z env:
2025-12-04T12:26:17.7031400Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.7031535Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:17.7031715Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:17.7031886Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:17.7032015Z ##[endgroup]
2025-12-04T12:26:17.7080760Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-12-04T12:26:17.7081068Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-12-04T12:26:17.7081271Z # Add render group for container creation.
2025-12-04T12:26:17.7081439Z render_gid=`cat /etc/group | grep render | cut -d: -f3`
2025-12-04T12:26:17.7081641Z # Ensure GPU isolation if pod is part of kubernetes setup with DEVICE_FLAG.
2025-12-04T12:26:17.7081842Z if [ -f "/etc/podinfo/gha-render-devices" ]; then
2025-12-04T12:26:17.7082025Z   DEVICE_FLAG=$(cat /etc/podinfo/gha-render-devices)
2025-12-04T12:26:17.7082162Z else
2025-12-04T12:26:17.7082263Z   DEVICE_FLAG="--device /dev/dri"
2025-12-04T12:26:17.7082375Z fi
2025-12-04T12:26:17.7082556Z # The --group-add daemon and --group-add bin are needed in the Ubuntu 24.04 and Almalinux OSs respectively.
2025-12-04T12:26:17.7082917Z # This is due to the device files (/dev/kfd & /dev/dri) being owned by video group on bare metal.
2025-12-04T12:26:17.7083167Z # This video group ID maps to subgid 1 inside the docker image due to the /etc/subgid entries.
2025-12-04T12:26:17.7083429Z # The group name corresponding to group ID 1 can change depending on the OS, so both are necessary.
2025-12-04T12:26:17.7083868Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd $DEVICE_FLAG --group-add video --group-add $render_gid --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host" >> "${GITHUB_ENV}"
2025-12-04T12:26:17.7087169Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:17.7087307Z env:
2025-12-04T12:26:17.7087397Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.7087528Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:17.7087698Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:17.7087862Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:17.7087981Z ##[endgroup]
2025-12-04T12:26:17.7147564Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722
2025-12-04T12:26:17.7147771Z with:
2025-12-04T12:26:17.7147918Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only
2025-12-04T12:26:17.7148095Z   aws-region: us-east-1
2025-12-04T12:26:17.7148213Z   role-duration-seconds: 18000
2025-12-04T12:26:17.7148333Z   audience: sts.amazonaws.com
2025-12-04T12:26:17.7148446Z env:
2025-12-04T12:26:17.7148535Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:17.7148759Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:17.7148936Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:17.7149096Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:17.7149595Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:17.7150086Z ##[endgroup]
2025-12-04T12:26:18.0298069Z Assuming role with OIDC
2025-12-04T12:26:18.3777760Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions
2025-12-04T12:26:18.4751694Z ##[group]Run aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076
2025-12-04T12:26:18.4751911Z with:
2025-12-04T12:26:18.4752020Z   mask-password: true
2025-12-04T12:26:18.4752156Z   registry-type: private
2025-12-04T12:26:18.4752273Z   skip-logout: false
2025-12-04T12:26:18.4752388Z env:
2025-12-04T12:26:18.4752491Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:18.4752642Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:18.4752839Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:18.4753023Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:18.4753588Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:18.4754130Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:18.4754260Z   AWS_REGION: us-east-1
2025-12-04T12:26:18.4754707Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:18.4754879Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:18.4757286Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:18.4757397Z ##[endgroup]
2025-12-04T12:26:18.8670577Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:19.4841907Z ##[group]Run env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:26:19.4842175Z env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:26:19.4842518Z env | grep '^CI' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:26:19.4842722Z env | grep '^RUNNER' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:26:19.4847386Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:19.4847547Z env:
2025-12-04T12:26:19.4847651Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:19.4847805Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:19.4848005Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:19.4848186Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:19.4848746Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:19.4849270Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:19.4849386Z   AWS_REGION: us-east-1
2025-12-04T12:26:19.4849571Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:19.4849728Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:19.4851790Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:19.4851897Z ##[endgroup]
2025-12-04T12:26:19.4947665Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-12-04T12:26:19.4947884Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-12-04T12:26:19.4948141Z if [[ $ngpu -lt 2 ]]; then #We are temporarily reducing this down to 2 from 4 so that we can run tests on nodes with less gpus.
2025-12-04T12:26:19.4948435Z   echo "Error: only $ngpu GPU(s) detected, at least 2 GPUs are needed for distributed jobs"
2025-12-04T12:26:19.4948623Z   exit 1
2025-12-04T12:26:19.4948720Z fi
2025-12-04T12:26:19.4952974Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:19.4953130Z env:
2025-12-04T12:26:19.4953231Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:19.4953391Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:19.4953580Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:19.4953757Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:19.4954293Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:19.4954792Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:19.4954915Z   AWS_REGION: us-east-1
2025-12-04T12:26:19.4955152Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:19.4955314Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:19.4957431Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:19.4957543Z ##[endgroup]
2025-12-04T12:26:19.6029440Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2025-12-04T12:26:19.6029642Z with:
2025-12-04T12:26:19.6029925Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:19.6030236Z   use-custom-docker-registry: true
2025-12-04T12:26:19.6030372Z   docker-build-dir: .ci/docker
2025-12-04T12:26:19.6030496Z   docker-build-script: ./build.sh
2025-12-04T12:26:19.6030620Z   working-directory: .
2025-12-04T12:26:19.6030767Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:19.6030924Z   force-push: false
2025-12-04T12:26:19.6031020Z env:
2025-12-04T12:26:19.6031116Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:19.6031254Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:19.6031429Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:19.6031605Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:19.6032120Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:19.6032696Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:19.6032815Z   AWS_REGION: us-east-1
2025-12-04T12:26:19.6032975Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:19.6033133Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:19.6035160Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:19.6035264Z ##[endgroup]
2025-12-04T12:26:19.6043688Z ##[group]Run set -ex
2025-12-04T12:26:19.6043816Z set -ex
2025-12-04T12:26:19.6043914Z 
2025-12-04T12:26:19.6044074Z # If the docker build directory or the build script doesn't exist, the action will
2025-12-04T12:26:19.6044327Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2025-12-04T12:26:19.6044552Z # job could then download the pre-built image as usual
2025-12-04T12:26:19.6044813Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then
2025-12-04T12:26:19.6045058Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6045193Z else
2025-12-04T12:26:19.6045307Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6045486Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6045643Z 
2025-12-04T12:26:19.6045847Z   echo "Not using custom ECR registry.  Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..."
2025-12-04T12:26:19.6046074Z   exit 0
2025-12-04T12:26:19.6046164Z fi
2025-12-04T12:26:19.6046248Z 
2025-12-04T12:26:19.6046382Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2025-12-04T12:26:19.6046613Z   # The docker image name already includes the ECR prefix and tag, so we can just
2025-12-04T12:26:19.6046856Z   # use it as it is, but first let's extract the tag
2025-12-04T12:26:19.6047040Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2025-12-04T12:26:19.6047232Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6047413Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6047562Z else
2025-12-04T12:26:19.6047672Z   if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then
2025-12-04T12:26:19.6047819Z     CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:}
2025-12-04T12:26:19.6047966Z     DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*}
2025-12-04T12:26:19.6048093Z   fi
2025-12-04T12:26:19.6048340Z   DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2025-12-04T12:26:19.6048568Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6048801Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6049052Z   echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6049208Z fi
2025-12-04T12:26:19.6052078Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:19.6052220Z env:
2025-12-04T12:26:19.6052310Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:19.6052442Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:19.6052616Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:19.6052777Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:19.6053277Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:19.6053812Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:19.6053924Z   AWS_REGION: us-east-1
2025-12-04T12:26:19.6054060Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:19.6054210Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:19.6056223Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:19.6056326Z   REPO_NAME: pytorch
2025-12-04T12:26:19.6056600Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:19.6056926Z   DOCKER_BUILD_DIR: .ci/docker
2025-12-04T12:26:19.6057042Z   DOCKER_BUILD_SCRIPT: ./build.sh
2025-12-04T12:26:19.6057190Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:19.6057349Z   USE_CUSTOM_DOCKER_REGISTRY: true
2025-12-04T12:26:19.6057464Z   CUSTOM_TAG_PREFIX: 
2025-12-04T12:26:19.6057567Z ##[endgroup]
2025-12-04T12:26:19.6076883Z + [[ -d .ci/docker ]]
2025-12-04T12:26:19.6077058Z + [[ -f .ci/docker/./build.sh ]]
2025-12-04T12:26:19.6077212Z + [[ true == \t\r\u\e ]]
2025-12-04T12:26:19.6077345Z + echo skip=false
2025-12-04T12:26:19.6077823Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2025-12-04T12:26:19.6085398Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:19.6085835Z ++ awk -F '[:,]' '{print $2}'
2025-12-04T12:26:19.6097679Z + DOCKER_TAG=pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:19.6098516Z + echo docker-tag=pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:19.6099502Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:19.6126269Z ##[group]Run set +e
2025-12-04T12:26:19.6126437Z set +e
2025-12-04T12:26:19.6126556Z set -x
2025-12-04T12:26:19.6126672Z 
2025-12-04T12:26:19.6126985Z login() {
2025-12-04T12:26:19.6127220Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-12-04T12:26:19.6127456Z }
2025-12-04T12:26:19.6127566Z 
2025-12-04T12:26:19.6127677Z retry () {
2025-12-04T12:26:19.6127818Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-12-04T12:26:19.6127972Z }
2025-12-04T12:26:19.6128080Z 
2025-12-04T12:26:19.6128204Z retry login "${DOCKER_REGISTRY}"
2025-12-04T12:26:19.6128352Z 
2025-12-04T12:26:19.6128600Z START_TIME=$(date +%s)
2025-12-04T12:26:19.6128756Z # Wait up to 120 minutes
2025-12-04T12:26:19.6128938Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do
2025-12-04T12:26:19.6129165Z   # Check if image already exists, if it does then skip building it
2025-12-04T12:26:19.6129389Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2025-12-04T12:26:19.6129557Z     exit 0
2025-12-04T12:26:19.6129672Z   fi
2025-12-04T12:26:19.6129778Z 
2025-12-04T12:26:19.6129953Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2025-12-04T12:26:19.6130220Z   # use this to differentiate between the Docker build and regular build jobs. For the
2025-12-04T12:26:19.6130464Z   # latter, it will wait for the Docker images to become available before continuing
2025-12-04T12:26:19.6130663Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2025-12-04T12:26:19.6130828Z     # It's a Docker build job, let's build the image
2025-12-04T12:26:19.6131077Z     break
2025-12-04T12:26:19.6131178Z   else
2025-12-04T12:26:19.6131316Z     # It's a regular build job, wait for the image to become available
2025-12-04T12:26:19.6131473Z     sleep 300
2025-12-04T12:26:19.6131576Z   fi
2025-12-04T12:26:19.6131670Z done
2025-12-04T12:26:19.6131762Z 
2025-12-04T12:26:19.6131903Z # NB: This part requires a full checkout. Otherwise, the merge base will
2025-12-04T12:26:19.6132114Z # be empty.  The default action would be to continue rebuild the image
2025-12-04T12:26:19.6132310Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2025-12-04T12:26:19.6132486Z   # if we're on the base branch then use the parent commit
2025-12-04T12:26:19.6132644Z   MERGE_BASE=$(git rev-parse HEAD~)
2025-12-04T12:26:19.6132772Z else
2025-12-04T12:26:19.6132906Z   # otherwise we're on a PR, so use the most recent base commit
2025-12-04T12:26:19.6133095Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2025-12-04T12:26:19.6133235Z fi
2025-12-04T12:26:19.6133326Z 
2025-12-04T12:26:19.6133427Z if [[ -z "${MERGE_BASE}" ]]; then
2025-12-04T12:26:19.6133571Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6133702Z 
2025-12-04T12:26:19.6133879Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2025-12-04T12:26:19.6134081Z   exit 0
2025-12-04T12:26:19.6134176Z fi
2025-12-04T12:26:19.6134263Z 
2025-12-04T12:26:19.6134409Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2025-12-04T12:26:19.6134661Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2025-12-04T12:26:19.6134873Z   exit 1
2025-12-04T12:26:19.6134963Z fi
2025-12-04T12:26:19.6135052Z 
2025-12-04T12:26:19.6135193Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2025-12-04T12:26:19.6135434Z # If no image exists but the hash is the same as the previous hash then we should error out here
2025-12-04T12:26:19.6135651Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2025-12-04T12:26:19.6135896Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2025-12-04T12:26:19.6136168Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2025-12-04T12:26:19.6136335Z fi
2025-12-04T12:26:19.6136421Z 
2025-12-04T12:26:19.6136527Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-12-04T12:26:19.6140753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:19.6140947Z env:
2025-12-04T12:26:19.6141041Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:19.6141183Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:19.6141357Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:19.6141522Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:19.6142021Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:19.6142507Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:19.6142622Z   AWS_REGION: us-east-1
2025-12-04T12:26:19.6142808Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:19.6142963Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:19.6144984Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:19.6145100Z   DOCKER_BUILD_DIR: .ci/docker
2025-12-04T12:26:19.6145241Z   BASE_REVISION: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:26:19.6145587Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:19.6146110Z   DOCKER_TAG: pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:19.6146339Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:19.6146490Z   DOCKER_PUSH: 
2025-12-04T12:26:19.6146584Z ##[endgroup]
2025-12-04T12:26:19.6162911Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:19.6163147Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:19.6165788Z + aws ecr get-login-password --region us-east-1
2025-12-04T12:26:19.6166033Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:19.6166522Z /home/runner/_work/_temp/b9d2b534-bf95-493a-ba98-53c46e1f0e38.sh: line 5: aws: command not found
2025-12-04T12:26:19.6257867Z Error: Cannot perform an interactive login from a non TTY device
2025-12-04T12:26:19.6268498Z + sleep 1
2025-12-04T12:26:20.6278414Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:20.6281370Z + aws ecr get-login-password --region us-east-1
2025-12-04T12:26:20.6281756Z /home/runner/_work/_temp/b9d2b534-bf95-493a-ba98-53c46e1f0e38.sh: line 5: aws: command not found
2025-12-04T12:26:20.6282384Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:20.6366732Z Error: Cannot perform an interactive login from a non TTY device
2025-12-04T12:26:20.6378098Z + sleep 2
2025-12-04T12:26:22.6390375Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:22.6393944Z + aws ecr get-login-password --region us-east-1
2025-12-04T12:26:22.6394411Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:22.6395004Z /home/runner/_work/_temp/b9d2b534-bf95-493a-ba98-53c46e1f0e38.sh: line 5: aws: command not found
2025-12-04T12:26:22.6472275Z Error: Cannot perform an interactive login from a non TTY device
2025-12-04T12:26:22.6486577Z ++ date +%s
2025-12-04T12:26:22.6493295Z + START_TIME=1764851182
2025-12-04T12:26:22.6497664Z ++ date +%s
2025-12-04T12:26:22.6505136Z + [[ 1764843982 -lt 1764851182 ]]
2025-12-04T12:26:22.6505665Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:23.9862477Z {
2025-12-04T12:26:23.9862875Z 	"schemaVersion": 2,
2025-12-04T12:26:23.9863329Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2025-12-04T12:26:23.9863714Z 	"config": {
2025-12-04T12:26:23.9864006Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2025-12-04T12:26:23.9864341Z 		"size": 30522,
2025-12-04T12:26:23.9864704Z 		"digest": "sha256:79498ef00fdf8abfcde955fd685c3a7412c33ca80383b5905abfdc3c70621215"
2025-12-04T12:26:23.9865785Z 	},
2025-12-04T12:26:23.9865967Z 	"layers": [
2025-12-04T12:26:23.9866161Z 		{
2025-12-04T12:26:23.9866449Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9866954Z 			"size": 30594402,
2025-12-04T12:26:23.9867319Z 			"digest": "sha256:02de03a7213b62b792ec66a7efb8c86c4117ca00fb8651facf8ecfe33044b485"
2025-12-04T12:26:23.9867704Z 		},
2025-12-04T12:26:23.9867872Z 		{
2025-12-04T12:26:23.9868142Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9868476Z 			"size": 1554,
2025-12-04T12:26:23.9868815Z 			"digest": "sha256:3a5718b5258e28918133dd74ea64bd506b2c15530a2fa8a72c45c5b0d8f7c7b0"
2025-12-04T12:26:23.9869189Z 		},
2025-12-04T12:26:23.9869354Z 		{
2025-12-04T12:26:23.9869625Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9869963Z 			"size": 335779211,
2025-12-04T12:26:23.9870307Z 			"digest": "sha256:bf3aa22776924a41b55849f0f30cb22af45d41da1177a9d682cf94cde99d8f98"
2025-12-04T12:26:23.9870630Z 		},
2025-12-04T12:26:23.9870768Z 		{
2025-12-04T12:26:23.9871153Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9871434Z 			"size": 704,
2025-12-04T12:26:23.9871720Z 			"digest": "sha256:9d58e5257cefd43e8226153d71d28a865253662146aa9fce9a9f95af67b497fa"
2025-12-04T12:26:23.9872036Z 		},
2025-12-04T12:26:23.9872177Z 		{
2025-12-04T12:26:23.9872408Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9872688Z 			"size": 1770,
2025-12-04T12:26:23.9872980Z 			"digest": "sha256:fde80a64553533a56c032d4bc388837e7d4631a0424d1bfe135703165b67fd4d"
2025-12-04T12:26:23.9873293Z 		},
2025-12-04T12:26:23.9873436Z 		{
2025-12-04T12:26:23.9873666Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9873948Z 			"size": 485,
2025-12-04T12:26:23.9874231Z 			"digest": "sha256:6931c5f20e80e481e4f484471ff3a02878b4f8c54a9a5a4717213fdaa35c0bff"
2025-12-04T12:26:23.9874543Z 		},
2025-12-04T12:26:23.9874693Z 		{
2025-12-04T12:26:23.9874926Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9875216Z 			"size": 120663474,
2025-12-04T12:26:23.9875525Z 			"digest": "sha256:170ea6d3edd62991e37d2e6ebe53dfcd4601f5d42e8f9720af5f8db5fc267856"
2025-12-04T12:26:23.9875850Z 		},
2025-12-04T12:26:23.9875993Z 		{
2025-12-04T12:26:23.9876221Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9876503Z 			"size": 4433,
2025-12-04T12:26:23.9876843Z 			"digest": "sha256:dc8487f6c81cac00fa33031f8d3481e2c3634c4f064a9c4c36b87b41e78bc9fb"
2025-12-04T12:26:23.9877166Z 		},
2025-12-04T12:26:23.9877309Z 		{
2025-12-04T12:26:23.9877540Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9877820Z 			"size": 1755,
2025-12-04T12:26:23.9878104Z 			"digest": "sha256:9748c5348f39a11c960c49fd9219fdea1c23e612ed11a02d71501424defc80f5"
2025-12-04T12:26:23.9878423Z 		},
2025-12-04T12:26:23.9878567Z 		{
2025-12-04T12:26:23.9878800Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9879086Z 			"size": 724,
2025-12-04T12:26:23.9879412Z 			"digest": "sha256:8539cc3f8d8a138501ed0255c0cd7ec491bc0add9e4a62095f1c0f9533daa1cc"
2025-12-04T12:26:23.9879736Z 		},
2025-12-04T12:26:23.9879872Z 		{
2025-12-04T12:26:23.9880048Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9880263Z 			"size": 3378352584,
2025-12-04T12:26:23.9880490Z 			"digest": "sha256:af88f886884fe6f1a1992efb7ce8473901f795eef69caa199443f3e076fdfd5b"
2025-12-04T12:26:23.9880728Z 		},
2025-12-04T12:26:23.9880834Z 		{
2025-12-04T12:26:23.9881007Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9881213Z 			"size": 396,
2025-12-04T12:26:23.9881425Z 			"digest": "sha256:32fbb88555c4195c45c7008cf92e389d67acc79a7e382503003ef93bcb886afe"
2025-12-04T12:26:23.9881844Z 		},
2025-12-04T12:26:23.9881984Z 		{
2025-12-04T12:26:23.9882915Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9883138Z 			"size": 80171601,
2025-12-04T12:26:23.9883359Z 			"digest": "sha256:3231e1ab814b143b244037c540b637be259085834865ac43b1ed2b6f6ad631e1"
2025-12-04T12:26:23.9883592Z 		},
2025-12-04T12:26:23.9883701Z 		{
2025-12-04T12:26:23.9883875Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9884085Z 			"size": 787,
2025-12-04T12:26:23.9884303Z 			"digest": "sha256:80061bf5dcbb9a4e38ac865a9cdc0a615bb294e3e6bfa357a6d515dcf3f54abc"
2025-12-04T12:26:23.9884546Z 		},
2025-12-04T12:26:23.9884655Z 		{
2025-12-04T12:26:23.9884827Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9885038Z 			"size": 106,
2025-12-04T12:26:23.9885254Z 			"digest": "sha256:6e9524f4518ec02b47ff12c55b6b6afbc65b3f4be59072e2afe20c2c87522549"
2025-12-04T12:26:23.9885495Z 		},
2025-12-04T12:26:23.9885603Z 		{
2025-12-04T12:26:23.9885780Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9885989Z 			"size": 1495,
2025-12-04T12:26:23.9886258Z 			"digest": "sha256:ce919d4bf5eeff71d49b160a16603117225530497c3905e02224227d11e2ff88"
2025-12-04T12:26:23.9886493Z 		},
2025-12-04T12:26:23.9886601Z 		{
2025-12-04T12:26:23.9886813Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9887024Z 			"size": 548601195,
2025-12-04T12:26:23.9887242Z 			"digest": "sha256:47681e3e6f37423139a5c86549ffbb43e4f258344b0461208f5821263da152e9"
2025-12-04T12:26:23.9887470Z 		},
2025-12-04T12:26:23.9887572Z 		{
2025-12-04T12:26:23.9887739Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9887943Z 			"size": 162,
2025-12-04T12:26:23.9888157Z 			"digest": "sha256:cb70fe22c9ebacebfe8402519059c8a66da6d5a77979e4c0ecdb3a762bebe357"
2025-12-04T12:26:23.9888399Z 		},
2025-12-04T12:26:23.9888501Z 		{
2025-12-04T12:26:23.9888666Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9888873Z 			"size": 104,
2025-12-04T12:26:23.9889082Z 			"digest": "sha256:17858e829c8cfe9a7e22516e03ad5273d8cf5c50f58edb10ff60c74e15c8e1f6"
2025-12-04T12:26:23.9889321Z 		},
2025-12-04T12:26:23.9889420Z 		{
2025-12-04T12:26:23.9889586Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9889793Z 			"size": 724,
2025-12-04T12:26:23.9890003Z 			"digest": "sha256:8539cc3f8d8a138501ed0255c0cd7ec491bc0add9e4a62095f1c0f9533daa1cc"
2025-12-04T12:26:23.9890235Z 		},
2025-12-04T12:26:23.9890336Z 		{
2025-12-04T12:26:23.9890484Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9890645Z 			"size": 196,
2025-12-04T12:26:23.9890814Z 			"digest": "sha256:a63f3b4eed1157bcb3c51b64196e74e9f10d1f923652b02fd433c6ed993597ff"
2025-12-04T12:26:23.9891000Z 		},
2025-12-04T12:26:23.9891082Z 		{
2025-12-04T12:26:23.9891215Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9891378Z 			"size": 2584,
2025-12-04T12:26:23.9891551Z 			"digest": "sha256:10ab3d1afbc4cb2d3ced8f3e0072c0b1dd124dcadcf68b95fadf8a7a9f663860"
2025-12-04T12:26:23.9891748Z 		},
2025-12-04T12:26:23.9891831Z 		{
2025-12-04T12:26:23.9891963Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9892176Z + exit 0
2025-12-04T12:26:23.9892269Z 			"size": 7652105336,
2025-12-04T12:26:23.9892445Z 			"digest": "sha256:98ca88b5095b449a2f2d753a21217856271912fbe51c2d99f928a2196f4097d5"
2025-12-04T12:26:23.9892633Z 		},
2025-12-04T12:26:23.9892720Z 		{
2025-12-04T12:26:23.9892857Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9893025Z 			"size": 135,
2025-12-04T12:26:23.9893195Z 			"digest": "sha256:025c90839a58c768b3cc444e48cae67c1a5b2c85320ad8827231f0ba390cf9aa"
2025-12-04T12:26:23.9893384Z 		},
2025-12-04T12:26:23.9893472Z 		{
2025-12-04T12:26:23.9893610Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9893778Z 			"size": 104,
2025-12-04T12:26:23.9893993Z 			"digest": "sha256:9255df5942ae69fee24f8074314f451d5d2f1ca71b6c777274297fd43a0032d8"
2025-12-04T12:26:23.9894179Z 		},
2025-12-04T12:26:23.9894265Z 		{
2025-12-04T12:26:23.9894403Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9894570Z 			"size": 612,
2025-12-04T12:26:23.9894736Z 			"digest": "sha256:f71ca9d4ed1c4ca8177602f3cb0db83d9787ea6c258a8ef203387b308ff3e0f0"
2025-12-04T12:26:23.9894922Z 		},
2025-12-04T12:26:23.9895004Z 		{
2025-12-04T12:26:23.9895136Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9895304Z 			"size": 838191953,
2025-12-04T12:26:23.9895481Z 			"digest": "sha256:d02b47b56ca7f3598f5943d4fdc7139d5e3d3bc82d49185cedf9817dd55fc75c"
2025-12-04T12:26:23.9895672Z 		},
2025-12-04T12:26:23.9895756Z 		{
2025-12-04T12:26:23.9895894Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9896061Z 			"size": 111,
2025-12-04T12:26:23.9896227Z 			"digest": "sha256:40279492aea7bc8fb650842b495912195621c21b14cef4c717a9e0a9fc535131"
2025-12-04T12:26:23.9896408Z 		},
2025-12-04T12:26:23.9896528Z 		{
2025-12-04T12:26:23.9896660Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9896866Z 			"size": 1556,
2025-12-04T12:26:23.9897036Z 			"digest": "sha256:33a27ce74abd7e32a03a564fc45005bc75904b53ad516f18d47facbeb2f2794e"
2025-12-04T12:26:23.9897221Z 		},
2025-12-04T12:26:23.9897302Z 		{
2025-12-04T12:26:23.9897434Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9897596Z 			"size": 107,
2025-12-04T12:26:23.9897764Z 			"digest": "sha256:6b66ed335d1d8df6140caba76d9c2babed83bb37962e1e638825d49e67184fa5"
2025-12-04T12:26:23.9897952Z 		},
2025-12-04T12:26:23.9898033Z 		{
2025-12-04T12:26:23.9898169Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9898334Z 			"size": 166,
2025-12-04T12:26:23.9898505Z 			"digest": "sha256:9f010fa04118bfee2d7b4481e6badb714032bde0652b04151a6599e57e1bd91b"
2025-12-04T12:26:23.9898699Z 		},
2025-12-04T12:26:23.9898788Z 		{
2025-12-04T12:26:23.9898932Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9899102Z 			"size": 3702493,
2025-12-04T12:26:23.9899282Z 			"digest": "sha256:6c64d5e8bb6ae6ef4e3f1d316429d8b14a6e8a1fb410fb83b96c8bbd4a0a095c"
2025-12-04T12:26:23.9899476Z 		},
2025-12-04T12:26:23.9899565Z 		{
2025-12-04T12:26:23.9899707Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9899875Z 			"size": 107,
2025-12-04T12:26:23.9900038Z 			"digest": "sha256:c20ea058f549f5f5538c95c5e0da23afbbc9fb7ffc1987d126fe684eeed743f5"
2025-12-04T12:26:23.9900219Z 		},
2025-12-04T12:26:23.9900301Z 		{
2025-12-04T12:26:23.9900429Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9900587Z 			"size": 829,
2025-12-04T12:26:23.9900745Z 			"digest": "sha256:3c4fd2d54638a1336d39769fe36041aa6d186a8dea0e7096b8d8a7068ba0d3c0"
2025-12-04T12:26:23.9900921Z 		},
2025-12-04T12:26:23.9901004Z 		{
2025-12-04T12:26:23.9901134Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9901298Z 			"size": 26673844,
2025-12-04T12:26:23.9901467Z 			"digest": "sha256:964ebac3d7a95c64ea7f0d828cd58e6244cc955e9a099a2525079ecf64026e3f"
2025-12-04T12:26:23.9901646Z 		},
2025-12-04T12:26:23.9901730Z 		{
2025-12-04T12:26:23.9901864Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9902022Z 			"size": 104,
2025-12-04T12:26:23.9902182Z 			"digest": "sha256:2aaa7210673fc5bd15d36e54ee5c3fb495d1eafa1cb8d686054ccedb1c37bfc8"
2025-12-04T12:26:23.9902363Z 		},
2025-12-04T12:26:23.9902447Z 		{
2025-12-04T12:26:23.9902577Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9902737Z 			"size": 424,
2025-12-04T12:26:23.9902899Z 			"digest": "sha256:fa273daa00371a98ed668535e14b8cc3cb425feba0b601b3e3c72314d0234312"
2025-12-04T12:26:23.9903082Z 		},
2025-12-04T12:26:23.9903172Z 		{
2025-12-04T12:26:23.9903350Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9903512Z 			"size": 19279582,
2025-12-04T12:26:23.9903685Z 			"digest": "sha256:d931a62fd2408369decfa0e6eac11768e35d0ffddee87d769c82aaf1ad7e2899"
2025-12-04T12:26:23.9903867Z 		},
2025-12-04T12:26:23.9903952Z 		{
2025-12-04T12:26:23.9904084Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9904242Z 			"size": 826,
2025-12-04T12:26:23.9904400Z 			"digest": "sha256:d3573d61c28e1400840260d3c2c786c9e104f6558162beac799e55b6f5c1e747"
2025-12-04T12:26:23.9904575Z 		},
2025-12-04T12:26:23.9904658Z 		{
2025-12-04T12:26:23.9904788Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9904947Z 			"size": 724,
2025-12-04T12:26:23.9905107Z 			"digest": "sha256:8539cc3f8d8a138501ed0255c0cd7ec491bc0add9e4a62095f1c0f9533daa1cc"
2025-12-04T12:26:23.9905283Z 		},
2025-12-04T12:26:23.9905367Z 		{
2025-12-04T12:26:23.9905499Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9905659Z 			"size": 149,
2025-12-04T12:26:23.9905887Z 			"digest": "sha256:f9b32f08c49055dd61bd359d5f42f6adb9e5a183c2821d97d11572dd7ce1e91f"
2025-12-04T12:26:23.9906066Z 		},
2025-12-04T12:26:23.9906149Z 		{
2025-12-04T12:26:23.9906279Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9906436Z 			"size": 136,
2025-12-04T12:26:23.9906593Z 			"digest": "sha256:3a0206399d60f6e8897f78c8e8f81b59d51969a329ef45485d28ae19607ca72c"
2025-12-04T12:26:23.9906807Z 		},
2025-12-04T12:26:23.9906888Z 		{
2025-12-04T12:26:23.9907013Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9907168Z 			"size": 140,
2025-12-04T12:26:23.9907326Z 			"digest": "sha256:386f322edd1c1c275126bab065c22fcd3950916c1fb8491a21a7f5c358af599a"
2025-12-04T12:26:23.9907503Z 		},
2025-12-04T12:26:23.9907585Z 		{
2025-12-04T12:26:23.9907715Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9907876Z 			"size": 32,
2025-12-04T12:26:23.9908047Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-12-04T12:26:23.9908232Z 		},
2025-12-04T12:26:23.9908314Z 		{
2025-12-04T12:26:23.9908445Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9908604Z 			"size": 223,
2025-12-04T12:26:23.9908764Z 			"digest": "sha256:bbe49df30697f6959cd958299909d9255cd54663ce2e9e2c2d378f8f9dfe8345"
2025-12-04T12:26:23.9908942Z 		},
2025-12-04T12:26:23.9909024Z 		{
2025-12-04T12:26:23.9909156Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9909322Z 			"size": 346,
2025-12-04T12:26:23.9909483Z 			"digest": "sha256:d6630aa6f375b12cb7471c5b60eb32e02ff8d70adf4497e061d6c15fead186c7"
2025-12-04T12:26:23.9909662Z 		},
2025-12-04T12:26:23.9909745Z 		{
2025-12-04T12:26:23.9909875Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9910033Z 			"size": 88302,
2025-12-04T12:26:23.9910201Z 			"digest": "sha256:6d807afc1309592c99c7d77af3874afb54c1718377fe721ac0cc616f59d291b9"
2025-12-04T12:26:23.9910387Z 		},
2025-12-04T12:26:23.9910472Z 		{
2025-12-04T12:26:23.9910602Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9910761Z 			"size": 106,
2025-12-04T12:26:23.9910921Z 			"digest": "sha256:60b679430e4e0b7690392dfe4f5dc417847f7a3ba2b761ce747b66d412e1d956"
2025-12-04T12:26:23.9911102Z 		},
2025-12-04T12:26:23.9911187Z 		{
2025-12-04T12:26:23.9911319Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9911477Z 			"size": 1671,
2025-12-04T12:26:23.9911641Z 			"digest": "sha256:3992ae84f9eda1c5c52fa96b1f1d0fc3f93c661c5cf0b971a504a260c290da49"
2025-12-04T12:26:23.9911821Z 		},
2025-12-04T12:26:23.9911900Z 		{
2025-12-04T12:26:23.9912030Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9912187Z 			"size": 724,
2025-12-04T12:26:23.9912394Z 			"digest": "sha256:8539cc3f8d8a138501ed0255c0cd7ec491bc0add9e4a62095f1c0f9533daa1cc"
2025-12-04T12:26:23.9912580Z 		},
2025-12-04T12:26:23.9912662Z 		{
2025-12-04T12:26:23.9912790Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9912947Z 			"size": 138,
2025-12-04T12:26:23.9913103Z 			"digest": "sha256:62d400609f9c38fce4745f72372423072ba0f142b3c03775ccb317f6c5240966"
2025-12-04T12:26:23.9913276Z 		},
2025-12-04T12:26:23.9913358Z 		{
2025-12-04T12:26:23.9913486Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9913641Z 			"size": 119,
2025-12-04T12:26:23.9925762Z 			"digest": "sha256:7e7b097490967d568331cc9f8afdd02422fe101c6364ec5e12dba2970991e533"
2025-12-04T12:26:23.9925960Z 		},
2025-12-04T12:26:23.9926049Z 		{
2025-12-04T12:26:23.9926184Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9926346Z 			"size": 6231259865,
2025-12-04T12:26:23.9926527Z 			"digest": "sha256:7dcdbd8421cb17aaa5d0cb965ddf94e196cb364e762b12ab78024cb25e3b6bcd"
2025-12-04T12:26:23.9926712Z 		},
2025-12-04T12:26:23.9926918Z 		{
2025-12-04T12:26:23.9927042Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9927196Z 			"size": 174,
2025-12-04T12:26:23.9927350Z 			"digest": "sha256:cbb12613719bab9f179968227f9fb8881251992804e460b9a9e1c00f3ac4a0c5"
2025-12-04T12:26:23.9927522Z 		},
2025-12-04T12:26:23.9927602Z 		{
2025-12-04T12:26:23.9927722Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9927873Z 			"size": 1896,
2025-12-04T12:26:23.9928030Z 			"digest": "sha256:e87038dce9bc8e13bd64006847d30ddcaf77455256c4985fccfec83f82d4b925"
2025-12-04T12:26:23.9928204Z 		},
2025-12-04T12:26:23.9928282Z 		{
2025-12-04T12:26:23.9928405Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9928559Z 			"size": 162783968,
2025-12-04T12:26:23.9928723Z 			"digest": "sha256:e4606b636f96f1c80f4be26aeb9d6f5f990f6149789c2de160451c5ac76a467d"
2025-12-04T12:26:23.9928897Z 		},
2025-12-04T12:26:23.9928976Z 		{
2025-12-04T12:26:23.9929108Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9929263Z 			"size": 302,
2025-12-04T12:26:23.9929418Z 			"digest": "sha256:6f2a5d33b946e561219b9968769773e36ce1d28bee8c62eff652098b7825fc79"
2025-12-04T12:26:23.9929587Z 		},
2025-12-04T12:26:23.9929663Z 		{
2025-12-04T12:26:23.9929794Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9929952Z 			"size": 32,
2025-12-04T12:26:23.9930113Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-12-04T12:26:23.9930289Z 		},
2025-12-04T12:26:23.9930368Z 		{
2025-12-04T12:26:23.9930493Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9930645Z 			"size": 108,
2025-12-04T12:26:23.9930800Z 			"digest": "sha256:a4f2bf2f19e63b91d46f2d9cf11a25c657517a6835996404da1e79a09d918b0e"
2025-12-04T12:26:23.9930980Z 		},
2025-12-04T12:26:23.9931062Z 		{
2025-12-04T12:26:23.9931187Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-12-04T12:26:23.9931345Z 			"size": 54145661,
2025-12-04T12:26:23.9931510Z 			"digest": "sha256:1ae00acdac56cbc6d3f81b3c5d854a2b77c30d458b0fbe18c5935145364484f0"
2025-12-04T12:26:23.9931686Z 		}
2025-12-04T12:26:23.9931764Z 	]
2025-12-04T12:26:23.9931845Z }
2025-12-04T12:26:23.9947234Z ##[group]Run set -eux
2025-12-04T12:26:23.9947359Z set -eux
2025-12-04T12:26:23.9947529Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have
2025-12-04T12:26:23.9947952Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true
2025-12-04T12:26:23.9952418Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:23.9952568Z env:
2025-12-04T12:26:23.9952668Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:23.9952863Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:23.9953051Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:23.9953222Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:23.9953729Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:23.9954223Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:23.9954347Z   AWS_REGION: us-east-1
2025-12-04T12:26:23.9954541Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:23.9954700Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:23.9957022Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:23.9957134Z ##[endgroup]
2025-12-04T12:26:23.9979359Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2025-12-04T12:26:23.9979766Z + jq --raw-output .SecretString
2025-12-04T12:26:23.9980069Z /home/runner/_work/_temp/be9808ab-d53c-4684-9f46-c94743e690f3.sh: line 3: aws: command not found
2025-12-04T12:26:23.9980656Z + jq -r .docker_hub_readonly_token
2025-12-04T12:26:23.9981820Z + docker login --username pytorchbot --password-stdin
2025-12-04T12:26:24.0070286Z Error: Cannot perform an interactive login from a non TTY device
2025-12-04T12:26:24.0077245Z + true
2025-12-04T12:26:24.0138884Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2025-12-04T12:26:24.0139079Z with:
2025-12-04T12:26:24.0139364Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:24.0139709Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:24.0139873Z env:
2025-12-04T12:26:24.0139978Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:24.0140129Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:24.0140317Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:24.0140510Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:24.0141048Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:24.0141556Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:24.0141686Z   AWS_REGION: us-east-1
2025-12-04T12:26:24.0141907Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:24.0142071Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:24.0144142Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:24.0144258Z ##[endgroup]
2025-12-04T12:26:24.0151278Z ##[group]Run set -x
2025-12-04T12:26:24.0151406Z set -x
2025-12-04T12:26:24.0151509Z set +e
2025-12-04T12:26:24.0151608Z 
2025-12-04T12:26:24.0151729Z login() {
2025-12-04T12:26:24.0151937Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-12-04T12:26:24.0152141Z }
2025-12-04T12:26:24.0152238Z 
2025-12-04T12:26:24.0152335Z retry () {
2025-12-04T12:26:24.0152461Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-12-04T12:26:24.0152601Z }
2025-12-04T12:26:24.0152697Z 
2025-12-04T12:26:24.0152804Z retry login "${DOCKER_REGISTRY}"
2025-12-04T12:26:24.0152932Z 
2025-12-04T12:26:24.0153131Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024')
2025-12-04T12:26:24.0153386Z echo "Compressed size of image in MB: ${IMAGE_SIZE}"
2025-12-04T12:26:24.0153539Z 
2025-12-04T12:26:24.0153633Z set -e
2025-12-04T12:26:24.0153781Z # ignore output since only exit code is used for conditional
2025-12-04T12:26:24.0153977Z # only pull docker image if it's not available locally
2025-12-04T12:26:24.0154198Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2025-12-04T12:26:24.0154396Z   retry docker pull "${DOCKER_IMAGE}"
2025-12-04T12:26:24.0154527Z fi
2025-12-04T12:26:24.0158875Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:24.0159030Z env:
2025-12-04T12:26:24.0159131Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:24.0159277Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:24.0159464Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:24.0159638Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:24.0160151Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:24.0160786Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:24.0160914Z   AWS_REGION: us-east-1
2025-12-04T12:26:24.0161058Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:24.0161223Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:24.0163264Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:24.0163628Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:24.0163956Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:24.0164117Z ##[endgroup]
2025-12-04T12:26:24.0179960Z + set +e
2025-12-04T12:26:24.0180110Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:24.0180286Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:24.0183162Z + aws ecr get-login-password --region us-east-1
2025-12-04T12:26:24.0183394Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:24.0183673Z /home/runner/_work/_temp/4cbeb1f6-9acd-4372-bea0-726639f85780.sh: line 5: aws: command not found
2025-12-04T12:26:24.0250443Z Error: Cannot perform an interactive login from a non TTY device
2025-12-04T12:26:24.0259162Z + sleep 1
2025-12-04T12:26:25.0268230Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:25.0272202Z + aws ecr get-login-password --region us-east-1
2025-12-04T12:26:25.0272881Z /home/runner/_work/_temp/4cbeb1f6-9acd-4372-bea0-726639f85780.sh: line 5: aws: command not found
2025-12-04T12:26:25.0273633Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:25.0362640Z Error: Cannot perform an interactive login from a non TTY device
2025-12-04T12:26:25.0373409Z + sleep 2
2025-12-04T12:26:27.0383512Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:27.0387756Z + aws ecr get-login-password --region us-east-1
2025-12-04T12:26:27.0388284Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T12:26:27.0388954Z /home/runner/_work/_temp/4cbeb1f6-9acd-4372-bea0-726639f85780.sh: line 5: aws: command not found
2025-12-04T12:26:27.0485980Z Error: Cannot perform an interactive login from a non TTY device
2025-12-04T12:26:27.0503803Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:27.0504411Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024'
2025-12-04T12:26:28.3972720Z + IMAGE_SIZE=18579.916069984436
2025-12-04T12:26:28.3973088Z + echo 'Compressed size of image in MB: 18579.916069984436'
2025-12-04T12:26:28.3973400Z + set -e
2025-12-04T12:26:28.3973953Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:26:28.3974550Z Compressed size of image in MB: 18579.916069984436
2025-12-04T12:26:28.4153808Z Prepare all required actions
2025-12-04T12:26:28.4170415Z ##[group]Run ./.github/actions/get-workflow-job-id
2025-12-04T12:26:28.4170691Z with:
2025-12-04T12:26:28.4171127Z   github-token: ***
2025-12-04T12:26:28.4171309Z env:
2025-12-04T12:26:28.4171486Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:28.4171746Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:28.4172094Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:28.4172416Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:28.4173404Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:28.4174366Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:28.4174612Z   AWS_REGION: us-east-1
2025-12-04T12:26:28.4175135Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:28.4175438Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:28.4179309Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:28.4179466Z ##[endgroup]
2025-12-04T12:26:28.4189433Z ##[group]Run set -eux
2025-12-04T12:26:28.4189567Z set -eux
2025-12-04T12:26:28.4189766Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-12-04T12:26:28.4194485Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:26:28.4194649Z env:
2025-12-04T12:26:28.4194754Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:28.4194908Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:28.4195111Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:28.4195303Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:28.4195884Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:28.4196454Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:28.4196589Z   AWS_REGION: us-east-1
2025-12-04T12:26:28.4196776Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:28.4196957Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:28.4199199Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:28.4199357Z   GITHUB_TOKEN: ***
2025-12-04T12:26:28.4199454Z ##[endgroup]
2025-12-04T12:26:28.4217882Z + python3 .github/scripts/get_workflow_job_id.py 19922798714 linux.rocm.gpu.gfx942.4.b-bphpw-runner-bfc27
2025-12-04T12:26:29.4080502Z Setting output job-id=57117547539
2025-12-04T12:26:29.4081379Z Setting output job-name=linux-noble-rocm-py3.12-mi300 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4.b, module:rocm, oncall:distributed, mem_leak_check)
2025-12-04T12:26:29.4179792Z Prepare all required actions
2025-12-04T12:26:29.4180010Z Getting action download info
2025-12-04T12:26:29.6634138Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2025-12-04T12:26:30.5622249Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093)
2025-12-04T12:26:31.4584156Z ##[group]Run ./.github/actions/download-build-artifacts
2025-12-04T12:26:31.4584324Z with:
2025-12-04T12:26:31.4584438Z   name: linux-noble-rocm-py3.12-mi300
2025-12-04T12:26:31.4584579Z   s3-bucket: gha-artifacts
2025-12-04T12:26:31.4584694Z env:
2025-12-04T12:26:31.4584793Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:31.4584934Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:31.4585119Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:31.4585291Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:31.4585833Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:31.4586348Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:31.4586469Z   AWS_REGION: us-east-1
2025-12-04T12:26:31.4586647Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:31.4586860Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:31.4588926Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:31.4589038Z ##[endgroup]
2025-12-04T12:26:31.4611046Z ##[group]Run seemethere/download-artifact-s3@v4
2025-12-04T12:26:31.4611235Z with:
2025-12-04T12:26:31.4611369Z   name: linux-noble-rocm-py3.12-mi300
2025-12-04T12:26:31.4611526Z   s3-bucket: gha-artifacts
2025-12-04T12:26:31.4611661Z   region: us-east-1
2025-12-04T12:26:31.4611780Z env:
2025-12-04T12:26:31.4611891Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:26:31.4612072Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:26:31.4612450Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:26:31.4612655Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:26:31.4613269Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:26:31.4613870Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:26:31.4614008Z   AWS_REGION: us-east-1
2025-12-04T12:26:31.4614247Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:26:31.4614433Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:26:31.4616735Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:26:31.4616902Z ##[endgroup]
2025-12-04T12:26:31.6823239Z (node:17073) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-12-04T12:26:31.6823639Z 
2025-12-04T12:26:31.6823814Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-12-04T12:26:31.6824221Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-12-04T12:26:31.6824633Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-12-04T12:26:31.9440198Z Found 1 objects with prefix pytorch/pytorch/19922798714/linux-noble-rocm-py3.12-mi300/
2025-12-04T12:26:31.9440937Z Starting download (1/1): /home/runner/_work/pytorch/pytorch/artifacts.zip
2025-12-04T12:27:57.5561441Z Finished download (1/1): /home/runner/_work/pytorch/pytorch/artifacts.zip
2025-12-04T12:27:57.5564690Z Artifact download has finished successfully
2025-12-04T12:27:57.5744071Z ##[group]Run unzip -o artifacts.zip
2025-12-04T12:27:57.5744267Z unzip -o artifacts.zip
2025-12-04T12:27:57.5749137Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:27:57.5749320Z env:
2025-12-04T12:27:57.5749651Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:27:57.5749827Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:27:57.5750060Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:27:57.5750263Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:27:57.5750851Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:27:57.5751355Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:27:57.5751475Z   AWS_REGION: us-east-1
2025-12-04T12:27:57.5751664Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:27:57.5751820Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:27:57.5753865Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:27:57.5753978Z ##[endgroup]
2025-12-04T12:27:57.5790483Z Archive:  artifacts.zip
2025-12-04T12:27:57.5791338Z    creating: dist/
2025-12-04T12:28:00.4677150Z   inflating: dist/torch-2.10.0a0+gitffd9b0f-cp312-cp312-linux_x86_64.whl  
2025-12-04T12:28:00.4756045Z   inflating: dist/.ninja_log         
2025-12-04T12:28:00.4756407Z    creating: build/custom_test_artifacts/
2025-12-04T12:28:00.4757070Z    creating: build/custom_test_artifacts/custom-op-build/
2025-12-04T12:28:00.4757499Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2025-12-04T12:28:00.4758007Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2025-12-04T12:28:00.4759453Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-12-04T12:28:00.4760010Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/
2025-12-04T12:28:00.4760543Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake  
2025-12-04T12:28:00.4761131Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/
2025-12-04T12:28:00.4761698Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/
2025-12-04T12:28:00.4762784Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c  
2025-12-04T12:28:00.4763429Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out  
2025-12-04T12:28:00.4764017Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake  
2025-12-04T12:28:00.4764504Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/
2025-12-04T12:28:00.4764974Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/
2025-12-04T12:28:00.4765529Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-12-04T12:28:00.4766093Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out  
2025-12-04T12:28:00.4766611Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake  
2025-12-04T12:28:00.4767230Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin  
2025-12-04T12:28:00.4767828Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin  
2025-12-04T12:28:00.4768382Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2025-12-04T12:28:00.4768792Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2025-12-04T12:28:00.4769244Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2025-12-04T12:28:00.4769696Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2025-12-04T12:28:00.4770196Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2025-12-04T12:28:00.4770950Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2025-12-04T12:28:00.4771485Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2025-12-04T12:28:00.4771989Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2025-12-04T12:28:00.4772501Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2025-12-04T12:28:00.4773017Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2025-12-04T12:28:00.4773535Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2025-12-04T12:28:00.4774044Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2025-12-04T12:28:00.4774431Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2025-12-04T12:28:00.4781296Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2025-12-04T12:28:00.4896725Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2025-12-04T12:28:00.4897131Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.d  
2025-12-04T12:28:00.4897474Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2025-12-04T12:28:00.4897837Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2025-12-04T12:28:00.4898246Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2025-12-04T12:28:00.4898627Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2025-12-04T12:28:00.4898986Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2025-12-04T12:28:00.4899360Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2025-12-04T12:28:00.4899867Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2025-12-04T12:28:00.4900240Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2025-12-04T12:28:00.4900612Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2025-12-04T12:28:00.4900984Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2025-12-04T12:28:00.4911544Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2025-12-04T12:28:00.4958825Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2025-12-04T12:28:00.4959200Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.d  
2025-12-04T12:28:00.4959534Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-12-04T12:28:00.4959861Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2025-12-04T12:28:00.4960164Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2025-12-04T12:28:00.4960436Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2025-12-04T12:28:00.4960704Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2025-12-04T12:28:00.4960978Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_outer_vec.cc  
2025-12-04T12:28:00.4961263Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_vec_ext.cc  
2025-12-04T12:28:00.4961949Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2025-12-04T12:28:00.4962309Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2025-12-04T12:28:00.4962556Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2025-12-04T12:28:00.5062965Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2025-12-04T12:28:00.5096511Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2025-12-04T12:28:00.5096788Z    creating: build/custom_test_artifacts/jit-hook-build/
2025-12-04T12:28:00.5096994Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2025-12-04T12:28:00.5097231Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2025-12-04T12:28:00.5099490Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-12-04T12:28:00.5099788Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/
2025-12-04T12:28:00.5100067Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake  
2025-12-04T12:28:00.5100353Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/
2025-12-04T12:28:00.5100628Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/
2025-12-04T12:28:00.5100963Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c  
2025-12-04T12:28:00.5102634Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out  
2025-12-04T12:28:00.5103428Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake  
2025-12-04T12:28:00.5104108Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/
2025-12-04T12:28:00.5104712Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/
2025-12-04T12:28:00.5105424Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-12-04T12:28:00.5106177Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out  
2025-12-04T12:28:00.5107277Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake  
2025-12-04T12:28:00.5108003Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin  
2025-12-04T12:28:00.5108750Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin  
2025-12-04T12:28:00.5109401Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2025-12-04T12:28:00.5109924Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2025-12-04T12:28:00.5110468Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2025-12-04T12:28:00.5111043Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2025-12-04T12:28:00.5111701Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2025-12-04T12:28:00.5112441Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2025-12-04T12:28:00.5113138Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2025-12-04T12:28:00.5113788Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2025-12-04T12:28:00.5114436Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2025-12-04T12:28:00.5114916Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2025-12-04T12:28:00.5115381Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2025-12-04T12:28:00.5115842Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2025-12-04T12:28:00.5116458Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2025-12-04T12:28:00.5121034Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2025-12-04T12:28:00.5157684Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2025-12-04T12:28:00.5158082Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.d  
2025-12-04T12:28:00.5158465Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-12-04T12:28:00.5158844Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2025-12-04T12:28:00.5159181Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2025-12-04T12:28:00.5159521Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2025-12-04T12:28:00.5159839Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2025-12-04T12:28:00.5160155Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_outer_vec.cc  
2025-12-04T12:28:00.5160470Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_vec_ext.cc  
2025-12-04T12:28:00.5161042Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2025-12-04T12:28:00.5162173Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2025-12-04T12:28:00.5162651Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2025-12-04T12:28:00.5184630Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2025-12-04T12:28:00.5184948Z    creating: build/custom_test_artifacts/custom-backend-build/
2025-12-04T12:28:00.5185254Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2025-12-04T12:28:00.5185609Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2025-12-04T12:28:00.5187502Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-12-04T12:28:00.5188383Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/
2025-12-04T12:28:00.5188781Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake  
2025-12-04T12:28:00.5189182Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/
2025-12-04T12:28:00.5189571Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/
2025-12-04T12:28:00.5190029Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c  
2025-12-04T12:28:00.5190492Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out  
2025-12-04T12:28:00.5190929Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake  
2025-12-04T12:28:00.5191325Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/
2025-12-04T12:28:00.5191716Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/
2025-12-04T12:28:00.5192173Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-12-04T12:28:00.5192678Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out  
2025-12-04T12:28:00.5193104Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake  
2025-12-04T12:28:00.5194007Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin  
2025-12-04T12:28:00.5194621Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin  
2025-12-04T12:28:00.5195040Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2025-12-04T12:28:00.5195566Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2025-12-04T12:28:00.5195926Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2025-12-04T12:28:00.5196306Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2025-12-04T12:28:00.5196720Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2025-12-04T12:28:00.5197228Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2025-12-04T12:28:00.5197667Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2025-12-04T12:28:00.5198088Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2025-12-04T12:28:00.5198527Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2025-12-04T12:28:00.5198982Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2025-12-04T12:28:00.5199423Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2025-12-04T12:28:00.5199854Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2025-12-04T12:28:00.5200274Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2025-12-04T12:28:00.5200745Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2025-12-04T12:28:00.5269306Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2025-12-04T12:28:00.5269643Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.d  
2025-12-04T12:28:00.5270136Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2025-12-04T12:28:00.5270759Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2025-12-04T12:28:00.5271171Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2025-12-04T12:28:00.5271565Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2025-12-04T12:28:00.5271931Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2025-12-04T12:28:00.5272318Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2025-12-04T12:28:00.5272699Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2025-12-04T12:28:00.5273085Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2025-12-04T12:28:00.5273476Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2025-12-04T12:28:00.5273854Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2025-12-04T12:28:00.5283923Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2025-12-04T12:28:00.5315777Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2025-12-04T12:28:00.5316328Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.d  
2025-12-04T12:28:00.5316711Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-12-04T12:28:00.5317217Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2025-12-04T12:28:00.5317537Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2025-12-04T12:28:00.5317847Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2025-12-04T12:28:00.5318147Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2025-12-04T12:28:00.5318450Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_outer_vec.cc  
2025-12-04T12:28:00.5318752Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_vec_ext.cc  
2025-12-04T12:28:00.5319112Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2025-12-04T12:28:00.5319542Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2025-12-04T12:28:00.5319813Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2025-12-04T12:28:00.5379408Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2025-12-04T12:28:00.5402688Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2025-12-04T12:28:00.5402905Z    creating: build/lib/
2025-12-04T12:28:00.5451578Z   inflating: build/lib/libprotobuf-lite.a  
2025-12-04T12:28:00.5714681Z   inflating: build/lib/libprotobuf.a  
2025-12-04T12:28:00.6008727Z   inflating: build/lib/libprotoc.a   
2025-12-04T12:28:00.6014423Z   inflating: build/lib/libpthreadpool.a  
2025-12-04T12:28:00.6019274Z   inflating: build/lib/libcpuinfo.a  
2025-12-04T12:28:00.6023594Z   inflating: build/lib/libcpuinfo_internals.a  
2025-12-04T12:28:00.6023968Z   inflating: build/lib/libclog.a     
2025-12-04T12:28:00.6035417Z   inflating: build/lib/libpytorch_qnnpack.a  
2025-12-04T12:28:00.6036327Z   inflating: build/lib/libnnpack_reference_layers.a  
2025-12-04T12:28:00.6147827Z   inflating: build/lib/libmicrokernels-prod.a  
2025-12-04T12:28:00.6158262Z   inflating: build/lib/libnnpack.a   
2025-12-04T12:28:00.6681441Z   inflating: build/lib/libmicrokernels-all.a  
2025-12-04T12:28:00.6722344Z   inflating: build/lib/libgtest.a    
2025-12-04T12:28:00.6732398Z   inflating: build/lib/libgmock.a    
2025-12-04T12:28:00.6732862Z   inflating: build/lib/libgtest_main.a  
2025-12-04T12:28:00.6733143Z   inflating: build/lib/libgmock_main.a  
2025-12-04T12:28:00.6786676Z   inflating: build/lib/libXNNPACK.a  
2025-12-04T12:28:00.6831710Z   inflating: build/lib/libbenchmark.a  
2025-12-04T12:28:00.6831979Z   inflating: build/lib/libbenchmark_main.a  
2025-12-04T12:28:00.6871307Z   inflating: build/lib/libasmjit.a   
2025-12-04T12:28:00.6871569Z   inflating: build/lib/libjitprofiling.a  
2025-12-04T12:28:00.6876161Z   inflating: build/lib/libittnotify.a  
2025-12-04T12:28:00.7563903Z   inflating: build/lib/libfbgemm.a   
2025-12-04T12:28:00.7581861Z   inflating: build/lib/libtensorpipe_uv.a  
2025-12-04T12:28:00.7902995Z   inflating: build/lib/libtensorpipe.a  
2025-12-04T12:28:00.7974793Z   inflating: build/lib/libgloo.a     
2025-12-04T12:28:00.8002392Z   inflating: build/lib/libonnx_proto.a  
2025-12-04T12:28:00.8248421Z   inflating: build/lib/libgloo_hip.a  
2025-12-04T12:28:00.8669794Z   inflating: build/lib/libonnx.a     
2025-12-04T12:28:01.4646906Z   inflating: build/lib/libdnnl.a     
2025-12-04T12:28:01.4658372Z   inflating: build/lib/libfmt.a      
2025-12-04T12:28:01.4842940Z   inflating: build/lib/libkineto.a   
2025-12-04T12:28:01.4912639Z   inflating: build/lib/libc10.so     
2025-12-04T12:28:01.4912991Z   inflating: build/lib/libtorch_global_deps.so  
2025-12-04T12:28:01.4913813Z   inflating: build/lib/libcaffe2_nvrtc.so  
2025-12-04T12:28:01.4940810Z   inflating: build/lib/libc10_hip.so  
2025-12-04T12:28:01.5224097Z   inflating: build/lib/libfbgemm_genai.a  
2025-12-04T12:28:03.3558024Z   inflating: build/lib/libtorch_cpu.so  
2025-12-04T12:28:03.3560253Z   inflating: build/lib/libshm.so     
2025-12-04T12:28:04.1963436Z   inflating: build/lib/libtorch_hip.so  
2025-12-04T12:28:04.1963947Z   inflating: build/lib/libtorch.so   
2025-12-04T12:28:04.1975281Z   inflating: build/lib/libjitbackend_test.so  
2025-12-04T12:28:04.1989019Z   inflating: build/lib/libbackend_with_compiler.so  
2025-12-04T12:28:04.2031357Z   inflating: build/lib/libtorchbind_test.so  
2025-12-04T12:28:04.2047125Z   inflating: build/lib/libaoti_custom_ops.so  
2025-12-04T12:28:04.3473887Z   inflating: build/lib/libtorch_python.so  
2025-12-04T12:28:04.3495768Z   inflating: build/lib/libnnapi_backend.so  
2025-12-04T12:28:04.3496090Z    creating: build/bin/
2025-12-04T12:28:04.3496334Z    creating: build/bin/CMakeFiles/
2025-12-04T12:28:04.3496609Z   inflating: build/bin/cmake_install.cmake  
2025-12-04T12:28:04.3497046Z   inflating: build/bin/CTestTestfile.cmake  
2025-12-04T12:28:04.3770712Z   inflating: build/bin/protoc-3.13.0.0  
2025-12-04T12:28:04.4045513Z   inflating: build/bin/protoc        
2025-12-04T12:28:04.4080420Z   inflating: build/bin/c10_AllocatorConfig_test  
2025-12-04T12:28:04.4113724Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2025-12-04T12:28:04.4147929Z   inflating: build/bin/c10_DeviceGuard_test  
2025-12-04T12:28:04.4182176Z   inflating: build/bin/c10_Device_test  
2025-12-04T12:28:04.4214707Z   inflating: build/bin/c10_StreamGuard_test  
2025-12-04T12:28:04.4250221Z   inflating: build/bin/c10_Scalar_test  
2025-12-04T12:28:04.4289336Z   inflating: build/bin/c10_DispatchKeySet_test  
2025-12-04T12:28:04.4325282Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2025-12-04T12:28:04.4362388Z   inflating: build/bin/c10_SymInt_test  
2025-12-04T12:28:04.4398951Z   inflating: build/bin/c10_InlineStreamGuard_test  
2025-12-04T12:28:04.4435957Z   inflating: build/bin/c10_SizesAndStrides_test  
2025-12-04T12:28:04.4468837Z   inflating: build/bin/c10_ArrayRef_test  
2025-12-04T12:28:04.4514479Z   inflating: build/bin/c10_cow_test  
2025-12-04T12:28:04.4547219Z   inflating: build/bin/c10_ConstexprCrc_test  
2025-12-04T12:28:04.4580053Z   inflating: build/bin/c10_DeadlockDetection_test  
2025-12-04T12:28:04.4617844Z   inflating: build/bin/c10_Enumerate_test  
2025-12-04T12:28:04.4652539Z   inflating: build/bin/c10_IntrusiveList_test  
2025-12-04T12:28:04.4686360Z   inflating: build/bin/c10_Half_test  
2025-12-04T12:28:04.4721268Z   inflating: build/bin/c10_Bitset_test  
2025-12-04T12:28:04.4758218Z   inflating: build/bin/c10_LeftRight_test  
2025-12-04T12:28:04.4791557Z   inflating: build/bin/c10_Semaphore_test  
2025-12-04T12:28:04.4826846Z   inflating: build/bin/c10_NetworkFlow_test  
2025-12-04T12:28:04.4863586Z   inflating: build/bin/c10_ThreadLocal_test  
2025-12-04T12:28:04.4896829Z   inflating: build/bin/c10_Synchronized_test  
2025-12-04T12:28:04.4931268Z   inflating: build/bin/c10_TypeIndex_test  
2025-12-04T12:28:04.4965396Z   inflating: build/bin/c10_accumulate_test  
2025-12-04T12:28:04.4998391Z   inflating: build/bin/c10_error_test  
2025-12-04T12:28:04.5031829Z   inflating: build/bin/c10_bit_cast_test  
2025-12-04T12:28:04.5068669Z   inflating: build/bin/c10_bfloat16_test  
2025-12-04T12:28:04.5107024Z   inflating: build/bin/c10_complex_test  
2025-12-04T12:28:04.5140512Z   inflating: build/bin/c10_exception_test  
2025-12-04T12:28:04.5177728Z   inflating: build/bin/c10_complex_math_test  
2025-12-04T12:28:04.5211469Z   inflating: build/bin/c10_flags_test  
2025-12-04T12:28:04.5244915Z   inflating: build/bin/c10_generic_math_test  
2025-12-04T12:28:04.5278777Z   inflating: build/bin/c10_irange_test  
2025-12-04T12:28:04.5376986Z   inflating: build/bin/c10_intrusive_ptr_test  
2025-12-04T12:28:04.5412446Z   inflating: build/bin/c10_lazy_test  
2025-12-04T12:28:04.5450088Z   inflating: build/bin/c10_logging_test  
2025-12-04T12:28:04.5483283Z   inflating: build/bin/c10_nofatal_test  
2025-12-04T12:28:04.5532017Z   inflating: build/bin/c10_optional_test  
2025-12-04T12:28:04.5567286Z   inflating: build/bin/c10_registry_test  
2025-12-04T12:28:04.5607791Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2025-12-04T12:28:04.5704086Z   inflating: build/bin/c10_small_vector_test  
2025-12-04T12:28:04.5738448Z   inflating: build/bin/c10_ssize_test  
2025-12-04T12:28:04.5775619Z   inflating: build/bin/c10_string_util_test  
2025-12-04T12:28:04.5808576Z   inflating: build/bin/c10_string_view_test  
2025-12-04T12:28:04.5837587Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2025-12-04T12:28:04.5870961Z   inflating: build/bin/c10_tempfile_test  
2025-12-04T12:28:04.5908087Z   inflating: build/bin/c10_typeid_test  
2025-12-04T12:28:04.5940819Z   inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test  
2025-12-04T12:28:04.5973388Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream  
2025-12-04T12:28:04.6005925Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device  
2025-12-04T12:28:04.6038497Z   inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes  
2025-12-04T12:28:04.6071084Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads  
2025-12-04T12:28:04.6103583Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks  
2025-12-04T12:28:04.6136117Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block  
2025-12-04T12:28:04.6168957Z   inflating: build/bin/c10_hip_HIPTest  
2025-12-04T12:28:04.6527795Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2025-12-04T12:28:04.6896122Z   inflating: build/bin/vec_test_all_types_AVX512  
2025-12-04T12:28:04.7269826Z   inflating: build/bin/vec_test_all_types_AVX2  
2025-12-04T12:28:04.7333257Z   inflating: build/bin/test_aoti_abi_check  
2025-12-04T12:28:04.7364911Z   inflating: build/bin/test_vec_half_DEFAULT  
2025-12-04T12:28:04.7398343Z   inflating: build/bin/test_vec_half_AVX2  
2025-12-04T12:28:04.7431367Z   inflating: build/bin/test_vec_half_AVX512  
2025-12-04T12:28:04.7466057Z   inflating: build/bin/BackoffTest   
2025-12-04T12:28:04.7501296Z   inflating: build/bin/FileStoreTest  
2025-12-04T12:28:04.7538577Z   inflating: build/bin/TCPStoreTest  
2025-12-04T12:28:04.7574155Z   inflating: build/bin/HashStoreTest  
2025-12-04T12:28:04.7617991Z   inflating: build/bin/ProcessGroupGlooTest  
2025-12-04T12:28:04.7619583Z   inflating: build/bin/example_allreduce  
2025-12-04T12:28:04.7621521Z   inflating: build/bin/torch_shm_manager  
2025-12-04T12:28:04.7657316Z   inflating: build/bin/static_runtime_bench  
2025-12-04T12:28:04.7813859Z   inflating: build/bin/static_runtime_test  
2025-12-04T12:28:04.7861172Z   inflating: build/bin/Dict_test     
2025-12-04T12:28:04.7895922Z   inflating: build/bin/Dimname_test  
2025-12-04T12:28:04.7938290Z   inflating: build/bin/MaybeOwned_test  
2025-12-04T12:28:04.7975675Z   inflating: build/bin/NamedTensor_test  
2025-12-04T12:28:04.8014284Z   inflating: build/bin/apply_utils_test  
2025-12-04T12:28:04.8052824Z   inflating: build/bin/atest         
2025-12-04T12:28:04.8094899Z   inflating: build/bin/basic         
2025-12-04T12:28:04.8130899Z   inflating: build/bin/broadcast_test  
2025-12-04T12:28:04.8164446Z   inflating: build/bin/cpu_allocator_test  
2025-12-04T12:28:04.8202741Z   inflating: build/bin/cpu_generator_test  
2025-12-04T12:28:04.8237676Z   inflating: build/bin/cpu_profiling_allocator_test  
2025-12-04T12:28:04.8297071Z   inflating: build/bin/cpu_rng_test  
2025-12-04T12:28:04.8331270Z   inflating: build/bin/dlconvertor_test  
2025-12-04T12:28:04.8369181Z   inflating: build/bin/extension_backend_test  
2025-12-04T12:28:04.8405883Z   inflating: build/bin/half_test     
2025-12-04T12:28:04.8468654Z   inflating: build/bin/ivalue_test   
2025-12-04T12:28:04.8501790Z   inflating: build/bin/lazy_tensor_test  
2025-12-04T12:28:04.8536840Z   inflating: build/bin/math_kernel_test  
2025-12-04T12:28:04.8571979Z   inflating: build/bin/memory_format_test  
2025-12-04T12:28:04.8607389Z   inflating: build/bin/memory_overlapping_test  
2025-12-04T12:28:04.8641150Z   inflating: build/bin/operator_name_test  
2025-12-04T12:28:04.8676439Z   inflating: build/bin/mobile_memory_cleanup  
2025-12-04T12:28:04.8713188Z   inflating: build/bin/native_test   
2025-12-04T12:28:04.8747885Z   inflating: build/bin/packedtensoraccessor_test  
2025-12-04T12:28:04.8781666Z   inflating: build/bin/operators_test  
2025-12-04T12:28:04.8825872Z   inflating: build/bin/pow_test      
2025-12-04T12:28:04.8863102Z   inflating: build/bin/quantized_test  
2025-12-04T12:28:04.8896977Z   inflating: build/bin/reportMemoryUsage_test  
2025-12-04T12:28:04.8930179Z   inflating: build/bin/reduce_ops_test  
2025-12-04T12:28:04.8964008Z   inflating: build/bin/StorageUtils_test  
2025-12-04T12:28:04.9002269Z   inflating: build/bin/scalar_test   
2025-12-04T12:28:04.9039113Z   inflating: build/bin/scalar_tensor_test  
2025-12-04T12:28:04.9073266Z   inflating: build/bin/stride_properties_test  
2025-12-04T12:28:04.9124556Z   inflating: build/bin/tensor_iterator_test  
2025-12-04T12:28:04.9160561Z   inflating: build/bin/test_parallel  
2025-12-04T12:28:04.9197002Z   inflating: build/bin/type_ptr_test  
2025-12-04T12:28:04.9230543Z   inflating: build/bin/thread_init_test  
2025-12-04T12:28:04.9265316Z   inflating: build/bin/undefined_tensor_test  
2025-12-04T12:28:04.9304060Z   inflating: build/bin/type_test     
2025-12-04T12:28:04.9337054Z   inflating: build/bin/verify_api_visibility  
2025-12-04T12:28:04.9370954Z   inflating: build/bin/weakref_test  
2025-12-04T12:28:04.9417231Z   inflating: build/bin/legacy_vmap_test  
2025-12-04T12:28:04.9451169Z   inflating: build/bin/wrapdim_test  
2025-12-04T12:28:04.9490091Z   inflating: build/bin/IListRef_test  
2025-12-04T12:28:04.9524156Z   inflating: build/bin/xla_tensor_test  
2025-12-04T12:28:04.9591201Z   inflating: build/bin/List_test     
2025-12-04T12:28:04.9667954Z   inflating: build/bin/kernel_function_legacy_test  
2025-12-04T12:28:04.9729271Z   inflating: build/bin/kernel_function_test  
2025-12-04T12:28:04.9772301Z   inflating: build/bin/KernelFunction_test  
2025-12-04T12:28:04.9852386Z   inflating: build/bin/kernel_lambda_legacy_test  
2025-12-04T12:28:04.9917405Z   inflating: build/bin/kernel_lambda_test  
2025-12-04T12:28:04.9978660Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2025-12-04T12:28:05.0018111Z   inflating: build/bin/kernel_stackbased_test  
2025-12-04T12:28:05.0052113Z   inflating: build/bin/CppSignature_test  
2025-12-04T12:28:05.0084680Z   inflating: build/bin/op_allowlist_test  
2025-12-04T12:28:05.0276316Z   inflating: build/bin/op_registration_test  
2025-12-04T12:28:05.0309147Z   inflating: build/bin/hip_complex_math_test  
2025-12-04T12:28:05.0352795Z   inflating: build/bin/inline_container_test  
2025-12-04T12:28:05.0389113Z   inflating: build/bin/backend_fallback_test  
2025-12-04T12:28:05.0423956Z   inflating: build/bin/hip_apply_test  
2025-12-04T12:28:05.0456962Z   inflating: build/bin/hip_complex_test  
2025-12-04T12:28:05.0489335Z   inflating: build/bin/hip_distributions_test  
2025-12-04T12:28:05.0521829Z   inflating: build/bin/hip_generator_test  
2025-12-04T12:28:05.0554351Z   inflating: build/bin/hip_half_test  
2025-12-04T12:28:05.0586908Z   inflating: build/bin/hip_integer_divider_test  
2025-12-04T12:28:05.0619282Z   inflating: build/bin/hip_optional_test  
2025-12-04T12:28:05.0651794Z   inflating: build/bin/hip_packedtensoraccessor_test  
2025-12-04T12:28:05.0686586Z   inflating: build/bin/hip_dlconvertor_test  
2025-12-04T12:28:05.0720338Z   inflating: build/bin/hip_vectorized_test  
2025-12-04T12:28:05.1394667Z   inflating: build/bin/test_jit      
2025-12-04T12:28:05.1608345Z   inflating: build/bin/test_lazy     
2025-12-04T12:28:05.1644824Z   inflating: build/bin/test_dist_autograd  
2025-12-04T12:28:05.1689930Z   inflating: build/bin/test_cpp_rpc  
2025-12-04T12:28:05.1690914Z   inflating: build/bin/parallel_benchmark  
2025-12-04T12:28:05.2410776Z   inflating: build/bin/test_api      
2025-12-04T12:28:05.2411191Z    creating: .additional_ci_files/
2025-12-04T12:28:05.2448924Z   inflating: .additional_ci_files/test-times.json  
2025-12-04T12:28:05.2589657Z   inflating: .additional_ci_files/test-class-times.json  
2025-12-04T12:28:05.2616540Z ##[group]Run rm artifacts.zip
2025-12-04T12:28:05.2616810Z rm artifacts.zip
2025-12-04T12:28:05.2621891Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:28:05.2622079Z env:
2025-12-04T12:28:05.2622194Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:05.2622356Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:05.2622555Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:05.2622744Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:05.2623307Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:05.2623858Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:05.2623986Z   AWS_REGION: us-east-1
2025-12-04T12:28:05.2624187Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:05.2624379Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:05.2626620Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:05.2626803Z ##[endgroup]
2025-12-04T12:28:05.3565507Z ##[group]Run df -H
2025-12-04T12:28:05.3565688Z df -H
2025-12-04T12:28:05.3570993Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:28:05.3571203Z env:
2025-12-04T12:28:05.3571323Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:05.3571477Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:05.3571681Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:05.3571869Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:05.3572426Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:05.3572972Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:05.3573116Z   AWS_REGION: us-east-1
2025-12-04T12:28:05.3573307Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:05.3573628Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:05.3575867Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:05.3575981Z ##[endgroup]
2025-12-04T12:28:05.3980961Z Filesystem      Size  Used Avail Use% Mounted on
2025-12-04T12:28:05.3981296Z overlay          16T  362G   15T   3% /
2025-12-04T12:28:05.3981566Z tmpfs            68M     0   68M   0% /dev
2025-12-04T12:28:05.3981828Z /dev/md0         16T  362G   15T   3% /run
2025-12-04T12:28:05.3982093Z shm              68M   17k   68M   1% /dev/shm
2025-12-04T12:28:05.3982611Z amdprj2-k8s_2   5.5T  120G  5.4T   3% /home/runner/pytorch-data
2025-12-04T12:28:05.3983018Z tmpfs           3.3T   13k  3.3T   1% /run/secrets/kubernetes.io/serviceaccount
2025-12-04T12:28:05.3983364Z tmpfs           1.7T     0  1.7T   0% /proc/acpi
2025-12-04T12:28:05.3983646Z tmpfs           1.7T     0  1.7T   0% /proc/scsi
2025-12-04T12:28:05.3983924Z tmpfs           1.7T     0  1.7T   0% /sys/firmware
2025-12-04T12:28:05.3984253Z tmpfs           1.7T     0  1.7T   0% /sys/devices/virtual/powercap
2025-12-04T12:28:05.4011038Z Prepare all required actions
2025-12-04T12:28:05.4011261Z Getting action download info
2025-12-04T12:28:05.7469486Z ##[group]Run ./.github/actions/download-td-artifacts
2025-12-04T12:28:05.7469653Z with:
2025-12-04T12:28:05.7469750Z env:
2025-12-04T12:28:05.7469846Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:05.7469985Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:05.7470164Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:05.7470340Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:05.7470853Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:05.7471350Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:05.7471497Z   AWS_REGION: us-east-1
2025-12-04T12:28:05.7471691Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:05.7471844Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:05.7473899Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:05.7474007Z ##[endgroup]
2025-12-04T12:28:05.7487416Z ##[group]Run seemethere/download-artifact-s3@v4
2025-12-04T12:28:05.7487547Z with:
2025-12-04T12:28:05.7487637Z   name: td_results
2025-12-04T12:28:05.7487737Z   s3-bucket: gha-artifacts
2025-12-04T12:28:05.7487844Z   region: us-east-1
2025-12-04T12:28:05.7487936Z env:
2025-12-04T12:28:05.7488027Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:05.7488157Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:05.7488335Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:05.7488502Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:05.7489013Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:05.7489508Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:05.7489628Z   AWS_REGION: us-east-1
2025-12-04T12:28:05.7489756Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:05.7489904Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:05.7491955Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:05.7492059Z ##[endgroup]
2025-12-04T12:28:05.9697287Z (node:17122) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-12-04T12:28:05.9697585Z 
2025-12-04T12:28:05.9697717Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-12-04T12:28:05.9698038Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-12-04T12:28:05.9698371Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-12-04T12:28:06.2415425Z Found 1 objects with prefix pytorch/pytorch/19922798714/td_results/
2025-12-04T12:28:06.2415984Z Starting download (1/1): /home/runner/_work/pytorch/pytorch/td_results.json
2025-12-04T12:28:06.6774119Z Finished download (1/1): /home/runner/_work/pytorch/pytorch/td_results.json
2025-12-04T12:28:06.6777939Z Artifact download has finished successfully
2025-12-04T12:28:06.6926357Z ##[group]Run mkdir -p .additional_ci_files
2025-12-04T12:28:06.6926611Z mkdir -p .additional_ci_files
2025-12-04T12:28:06.6926933Z mv td_results.json .additional_ci_files/td_results.json || true
2025-12-04T12:28:06.6931913Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:28:06.6932094Z env:
2025-12-04T12:28:06.6932211Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:06.6932370Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:06.6932582Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:06.6932778Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:06.6933545Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:06.6934121Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:06.6934261Z   AWS_REGION: us-east-1
2025-12-04T12:28:06.6934559Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:06.6934741Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:06.6937294Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:06.6937425Z ##[endgroup]
2025-12-04T12:28:06.7001593Z ##[group]Run .github/scripts/parse_ref.py
2025-12-04T12:28:06.7001782Z .github/scripts/parse_ref.py
2025-12-04T12:28:06.7005533Z shell: /usr/bin/bash -e {0}
2025-12-04T12:28:06.7005676Z env:
2025-12-04T12:28:06.7005789Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:06.7005963Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:06.7006198Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:06.7006413Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:06.7007129Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:06.7007763Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:06.7007906Z   AWS_REGION: us-east-1
2025-12-04T12:28:06.7008096Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:06.7008292Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:06.7010886Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:06.7010992Z ##[endgroup]
2025-12-04T12:28:06.7114362Z Setting output branch=main
2025-12-04T12:28:06.7185771Z Prepare all required actions
2025-12-04T12:28:06.7186037Z Getting action download info
2025-12-04T12:28:06.9173864Z ##[group]Run ./.github/actions/filter-test-configs
2025-12-04T12:28:06.9174012Z with:
2025-12-04T12:28:06.9174257Z   github-token: ***
2025-12-04T12:28:06.9175549Z   test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}]}
2025-12-04T12:28:06.9177389Z   job-name: linux-noble-rocm-py3.12-mi300 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4.b, module:rocm, oncall:distributed, mem_leak_check)
2025-12-04T12:28:06.9177642Z env:
2025-12-04T12:28:06.9177741Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:06.9177886Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:06.9178072Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:06.9178245Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:06.9178754Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:06.9179248Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:06.9179371Z   AWS_REGION: us-east-1
2025-12-04T12:28:06.9179501Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:06.9179660Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:06.9181709Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:06.9181821Z ##[endgroup]
2025-12-04T12:28:06.9196835Z ##[group]Run nick-fields/retry@v3.0.0
2025-12-04T12:28:06.9196959Z with:
2025-12-04T12:28:06.9197046Z   shell: bash
2025-12-04T12:28:06.9197140Z   timeout_minutes: 10
2025-12-04T12:28:06.9197240Z   max_attempts: 5
2025-12-04T12:28:06.9197338Z   retry_wait_seconds: 30
2025-12-04T12:28:06.9197631Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.2

2025-12-04T12:28:06.9197934Z   polling_interval_seconds: 1
2025-12-04T12:28:06.9198049Z   warning_on_retry: true
2025-12-04T12:28:06.9198157Z   continue_on_error: false
2025-12-04T12:28:06.9198260Z env:
2025-12-04T12:28:06.9198350Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:06.9198482Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:06.9198663Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:06.9198831Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:06.9199336Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:06.9199903Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:06.9200019Z   AWS_REGION: us-east-1
2025-12-04T12:28:06.9200146Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:06.9200300Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:06.9202342Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:06.9202492Z   GITHUB_TOKEN: ***
2025-12-04T12:28:06.9202588Z ##[endgroup]
2025-12-04T12:28:06.9590532Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2
2025-12-04T12:28:07.1006611Z Defaulting to user installation because normal site-packages is not writeable
2025-12-04T12:28:07.1968045Z Collecting requests==2.27.1
2025-12-04T12:28:07.2291131Z   Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
2025-12-04T12:28:07.2390824Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 KB 6.3 MB/s eta 0:00:00
2025-12-04T12:28:07.2844915Z Collecting pyyaml==6.0.2
2025-12-04T12:28:07.2951954Z   Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB)
2025-12-04T12:28:07.3365202Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 751.2/751.2 KB 18.6 MB/s eta 0:00:00
2025-12-04T12:28:07.3686019Z Collecting urllib3<1.27,>=1.21.1
2025-12-04T12:28:07.3746017Z   Downloading urllib3-1.26.20-py2.py3-none-any.whl (144 kB)
2025-12-04T12:28:07.3805208Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 144.2/144.2 KB 26.9 MB/s eta 0:00:00
2025-12-04T12:28:07.3982960Z Collecting certifi>=2017.4.17
2025-12-04T12:28:07.4032740Z   Downloading certifi-2025.11.12-py3-none-any.whl (159 kB)
2025-12-04T12:28:07.4107785Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 159.4/159.4 KB 22.8 MB/s eta 0:00:00
2025-12-04T12:28:07.4992948Z Collecting charset-normalizer~=2.0.0
2025-12-04T12:28:07.5044321Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2025-12-04T12:28:07.5186572Z Collecting idna<4,>=2.5
2025-12-04T12:28:07.5236638Z   Downloading idna-3.11-py3-none-any.whl (71 kB)
2025-12-04T12:28:07.5264257Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.0/71.0 KB 33.0 MB/s eta 0:00:00
2025-12-04T12:28:07.5825357Z Installing collected packages: urllib3, pyyaml, idna, charset-normalizer, certifi, requests
2025-12-04T12:28:07.6749027Z   WARNING: The script normalizer is installed in '/home/runner/.local/bin' which is not on PATH.
2025-12-04T12:28:07.6749384Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2025-12-04T12:28:07.6919894Z Successfully installed certifi-2025.11.12 charset-normalizer-2.0.12 idna-3.11 pyyaml-6.0.2 requests-2.27.1 urllib3-1.26.20
2025-12-04T12:28:07.9582843Z Command completed after 1 attempt(s).
2025-12-04T12:28:07.9633516Z ##[group]Run set -x
2025-12-04T12:28:07.9633687Z set -x
2025-12-04T12:28:07.9633813Z 
2025-12-04T12:28:07.9634012Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-12-04T12:28:07.9634263Z # in runner workspace
2025-12-04T12:28:07.9634472Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2025-12-04T12:28:07.9639934Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:28:07.9640136Z env:
2025-12-04T12:28:07.9640264Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:07.9640453Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:07.9640697Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:07.9640913Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:07.9641530Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:07.9642055Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:07.9642181Z   AWS_REGION: us-east-1
2025-12-04T12:28:07.9642359Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:07.9642518Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:07.9644672Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:07.9644905Z ##[endgroup]
2025-12-04T12:28:07.9664019Z + python3 /home/runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2025-12-04T12:28:07.9749466Z Setting output branch=main
2025-12-04T12:28:07.9785685Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2025-12-04T12:28:07.9785928Z echo "Workflow: ${GITHUB_WORKFLOW}"
2025-12-04T12:28:07.9786109Z echo "Job name: ${JOB_NAME}"
2025-12-04T12:28:07.9786272Z 
2025-12-04T12:28:07.9786480Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-12-04T12:28:07.9786811Z # in runner workspace
2025-12-04T12:28:07.9787035Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2025-12-04T12:28:07.9787289Z   --workflow "${GITHUB_WORKFLOW}" \
2025-12-04T12:28:07.9787482Z   --job-name "${JOB_NAME}" \
2025-12-04T12:28:07.9789280Z   --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}]}" \
2025-12-04T12:28:07.9791113Z   --selected-test-configs "" \
2025-12-04T12:28:07.9791259Z   --pr-number "${PR_NUMBER}" \
2025-12-04T12:28:07.9791398Z   --tag "${TAG}" \
2025-12-04T12:28:07.9791531Z   --event-name "${EVENT_NAME}" \
2025-12-04T12:28:07.9791667Z   --schedule "${SCHEDULE}" \
2025-12-04T12:28:07.9791803Z   --branch "${HEAD_BRANCH}"
2025-12-04T12:28:07.9795870Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:28:07.9796038Z env:
2025-12-04T12:28:07.9796143Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:07.9796289Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:07.9796476Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:07.9796662Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:07.9797387Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:07.9797912Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:07.9798042Z   AWS_REGION: us-east-1
2025-12-04T12:28:07.9798227Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:07.9798394Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:07.9800552Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:07.9800773Z   GITHUB_TOKEN: ***
2025-12-04T12:28:07.9801022Z   JOB_NAME: linux-noble-rocm-py3.12-mi300 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4.b, module:rocm, oncall:distributed, mem_leak_check)
2025-12-04T12:28:07.9801275Z   PR_NUMBER: 
2025-12-04T12:28:07.9801371Z   TAG: 
2025-12-04T12:28:07.9801463Z   EVENT_NAME: schedule
2025-12-04T12:28:07.9801571Z   SCHEDULE: 29 8 * * *
2025-12-04T12:28:07.9801676Z   HEAD_BRANCH: main
2025-12-04T12:28:07.9801785Z ##[endgroup]
2025-12-04T12:28:07.9822439Z Workflow: periodic-rocm-mi300
2025-12-04T12:28:07.9822721Z Job name: linux-noble-rocm-py3.12-mi300 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4.b, module:rocm, oncall:distributed, mem_leak_check)
2025-12-04T12:28:08.5986047Z Setting output keep-going=True
2025-12-04T12:28:08.5986435Z Setting output ci-verbose-test-logs=False
2025-12-04T12:28:08.5986963Z Setting output ci-test-showlocals=False
2025-12-04T12:28:08.5987290Z Setting output ci-no-test-timeout=False
2025-12-04T12:28:08.5987597Z Setting output ci-no-td=False
2025-12-04T12:28:08.5987878Z Setting output ci-td-distributed=False
2025-12-04T12:28:08.5988183Z Setting output is-unstable=False
2025-12-04T12:28:08.5988475Z Setting output reenabled-issues=
2025-12-04T12:28:08.5994817Z Setting output test-matrix={"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}]}
2025-12-04T12:28:08.6001390Z Setting output is-test-matrix-empty=False
2025-12-04T12:28:08.6059697Z ##[group]Run echo "Filtered matrix:"
2025-12-04T12:28:08.6059990Z echo "Filtered matrix:"
2025-12-04T12:28:08.6063895Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.gfx942.4.b", "owners": ["module:rocm", "oncall:distributed"], "rerun_disabled_tests": "rerun_disabled_tests"}]}"
2025-12-04T12:28:08.6067943Z 
2025-12-04T12:28:08.6068065Z echo
2025-12-04T12:28:08.6068233Z echo "Is the current job unstable? False"
2025-12-04T12:28:08.6068421Z 
2025-12-04T12:28:08.6068545Z echo
2025-12-04T12:28:08.6068689Z echo "Is keep-going label set? True"
2025-12-04T12:28:08.6068874Z 
2025-12-04T12:28:08.6068988Z echo
2025-12-04T12:28:08.6069130Z echo "Reenabled issues? "
2025-12-04T12:28:08.6073434Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:28:08.6073594Z env:
2025-12-04T12:28:08.6073701Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:08.6073853Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:08.6074047Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:08.6074229Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:08.6074784Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:08.6075324Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:08.6075456Z   AWS_REGION: us-east-1
2025-12-04T12:28:08.6075625Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:08.6075856Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:08.6078278Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:08.6078400Z ##[endgroup]
2025-12-04T12:28:08.6109416Z Filtered matrix:
2025-12-04T12:28:08.6113538Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], mem_leak_check: mem_leak_check}, {config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], mem_leak_check: mem_leak_check}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], mem_leak_check: mem_leak_check}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu.gfx942.4.b, owners: [module:rocm, oncall:distributed], rerun_disabled_tests: rerun_disabled_tests}]}
2025-12-04T12:28:08.6117419Z 
2025-12-04T12:28:08.6117496Z Is the current job unstable? False
2025-12-04T12:28:08.6117606Z 
2025-12-04T12:28:08.6117676Z Is keep-going label set? True
2025-12-04T12:28:08.6117780Z 
2025-12-04T12:28:08.6117842Z Reenabled issues? 
2025-12-04T12:28:08.6146049Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-12-04T12:28:08.6146275Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-12-04T12:28:08.6150450Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:28:08.6150604Z env:
2025-12-04T12:28:08.6150704Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:08.6150847Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:08.6151026Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:08.6151199Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:08.6151704Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:08.6152218Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:08.6152341Z   AWS_REGION: us-east-1
2025-12-04T12:28:08.6152511Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:08.6152670Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:08.6154693Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:08.6154802Z   JOB_TIMEOUT: 600
2025-12-04T12:28:08.6154907Z ##[endgroup]
2025-12-04T12:28:08.6200382Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:28:08.6200645Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:28:08.6200859Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-12-04T12:28:08.6205473Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T12:28:08.6205660Z env:
2025-12-04T12:28:08.6205777Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:08.6205950Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:08.6206162Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:08.6206364Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:08.6207031Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:08.6207618Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:08.6207765Z   AWS_REGION: us-east-1
2025-12-04T12:28:08.6207974Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:08.6208164Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:08.6210451Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:08.6210571Z ##[endgroup]
2025-12-04T12:28:08.6281998Z ##[group]Run set -x
2025-12-04T12:28:08.6282144Z set -x
2025-12-04T12:28:08.6282243Z 
2025-12-04T12:28:08.6282352Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2025-12-04T12:28:08.6282514Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2025-12-04T12:28:08.6282678Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2025-12-04T12:28:08.6282827Z   TEST_COMMAND=.ci/caffe2/test.sh
2025-12-04T12:28:08.6282956Z else
2025-12-04T12:28:08.6283196Z   TEST_COMMAND=.ci/pytorch/test.sh
2025-12-04T12:28:08.6283318Z fi
2025-12-04T12:28:08.6283413Z 
2025-12-04T12:28:08.6283552Z # detached container should get cleaned up by teardown_ec2_linux
2025-12-04T12:28:08.6283759Z # TODO: Stop building test binaries as part of the build phase
2025-12-04T12:28:08.6283953Z # Used for GPU_FLAG since that doesn't play nice
2025-12-04T12:28:08.6284127Z # shellcheck disable=SC2086,SC2090
2025-12-04T12:28:08.6284269Z container_name=$(docker run \
2025-12-04T12:28:08.6284400Z   ${GPU_FLAG:-} \
2025-12-04T12:28:08.6284521Z   -e BUILD_ENVIRONMENT \
2025-12-04T12:28:08.6284646Z   -e PR_NUMBER \
2025-12-04T12:28:08.6284763Z   -e GITHUB_ACTIONS \
2025-12-04T12:28:08.6284882Z   -e GITHUB_REPOSITORY \
2025-12-04T12:28:08.6285006Z   -e GITHUB_WORKFLOW \
2025-12-04T12:28:08.6285122Z   -e GITHUB_JOB \
2025-12-04T12:28:08.6285240Z   -e GITHUB_RUN_ID \
2025-12-04T12:28:08.6285356Z   -e GITHUB_RUN_NUMBER \
2025-12-04T12:28:08.6285479Z   -e GITHUB_RUN_ATTEMPT \
2025-12-04T12:28:08.6285598Z   -e JOB_ID \
2025-12-04T12:28:08.6285703Z   -e JOB_NAME \
2025-12-04T12:28:08.6285808Z   -e BASE_SHA \
2025-12-04T12:28:08.6285910Z   -e BRANCH \
2025-12-04T12:28:08.6286008Z   -e SHA1 \
2025-12-04T12:28:08.6286113Z   -e AWS_DEFAULT_REGION \
2025-12-04T12:28:08.6286231Z   -e IN_WHEEL_TEST \
2025-12-04T12:28:08.6286343Z   -e SHARD_NUMBER \
2025-12-04T12:28:08.6286453Z   -e TEST_CONFIG \
2025-12-04T12:28:08.6286565Z   -e NUM_TEST_SHARDS \
2025-12-04T12:28:08.6286681Z   -e REENABLED_ISSUES \
2025-12-04T12:28:08.6286943Z   -e CONTINUE_THROUGH_ERROR \
2025-12-04T12:28:08.6287068Z   -e VERBOSE_TEST_LOGS \
2025-12-04T12:28:08.6287188Z   -e TEST_SHOWLOCALS \
2025-12-04T12:28:08.6287302Z   -e NO_TEST_TIMEOUT \
2025-12-04T12:28:08.6287409Z   -e NO_TD \
2025-12-04T12:28:08.6287523Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2025-12-04T12:28:08.6287667Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2025-12-04T12:28:08.6287812Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2025-12-04T12:28:08.6287942Z   -e TESTS_TO_INCLUDE \
2025-12-04T12:28:08.6288059Z   -e HUGGING_FACE_HUB_TOKEN \
2025-12-04T12:28:08.6288182Z   -e DASHBOARD_TAG \
2025-12-04T12:28:08.6288326Z   --env-file="${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" \
2025-12-04T12:28:08.6288485Z   --ulimit stack=10485760:83886080 \
2025-12-04T12:28:08.6288606Z   --ulimit core=0 \
2025-12-04T12:28:08.6288737Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2025-12-04T12:28:08.6288889Z   --security-opt seccomp=unconfined \
2025-12-04T12:28:08.6289021Z   --cap-add=SYS_PTRACE \
2025-12-04T12:28:08.6289146Z   --shm-size="8g" \
2025-12-04T12:28:08.6289264Z   --tty \
2025-12-04T12:28:08.6289362Z   --detach \
2025-12-04T12:28:08.6289472Z   --name="${container_name}" \
2025-12-04T12:28:08.6289595Z   --user jenkins \
2025-12-04T12:28:08.6289735Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2025-12-04T12:28:08.6289891Z   -w /var/lib/jenkins/workspace \
2025-12-04T12:28:08.6290078Z   "${DOCKER_IMAGE}"
2025-12-04T12:28:08.6290182Z )
2025-12-04T12:28:08.6290284Z # save container name for later step
2025-12-04T12:28:08.6290443Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV"
2025-12-04T12:28:08.6290710Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home
2025-12-04T12:28:08.6291053Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}"
2025-12-04T12:28:08.6294253Z shell: /usr/bin/bash -e {0}
2025-12-04T12:28:08.6294359Z env:
2025-12-04T12:28:08.6294447Z   GIT_DEFAULT_BRANCH: main
2025-12-04T12:28:08.6294580Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T12:28:08.6294754Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T12:28:08.6294915Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T12:28:08.6295415Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T12:28:08.6295901Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T12:28:08.6296015Z   AWS_REGION: us-east-1
2025-12-04T12:28:08.6296150Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T12:28:08.6296299Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T12:28:08.6298368Z   AWS_SESSION_TOKEN: ***
2025-12-04T12:28:08.6298496Z   BUILD_ENVIRONMENT: linux-noble-rocm-py3.12-mi300
2025-12-04T12:28:08.6298628Z   PR_NUMBER: 
2025-12-04T12:28:08.6298729Z   GITHUB_REPOSITORY: pytorch/pytorch
2025-12-04T12:28:08.6298855Z   GITHUB_WORKFLOW: periodic-rocm-mi300
2025-12-04T12:28:08.6298974Z   GITHUB_JOB: test
2025-12-04T12:28:08.6299071Z   GITHUB_RUN_ID: 19922798714
2025-12-04T12:28:08.6299176Z   GITHUB_RUN_NUMBER: 1861
2025-12-04T12:28:08.6299282Z   GITHUB_RUN_ATTEMPT: 1
2025-12-04T12:28:08.6299383Z   JOB_ID: 57117547539
2025-12-04T12:28:08.6299617Z   JOB_NAME: linux-noble-rocm-py3.12-mi300 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4.b, module:rocm, oncall:distributed, mem_leak_check)
2025-12-04T12:28:08.6299857Z   BRANCH: main
2025-12-04T12:28:08.6299963Z   SHA1: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:08.6300115Z   BASE_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:08.6300247Z   TEST_CONFIG: distributed
2025-12-04T12:28:08.6300352Z   SHARD_NUMBER: 3
2025-12-04T12:28:08.6300443Z   NUM_TEST_SHARDS: 3
2025-12-04T12:28:08.6300540Z   REENABLED_ISSUES: 
2025-12-04T12:28:08.6300637Z   CONTINUE_THROUGH_ERROR: True
2025-12-04T12:28:08.6300747Z   VERBOSE_TEST_LOGS: False
2025-12-04T12:28:08.6300852Z   TEST_SHOWLOCALS: False
2025-12-04T12:28:08.6300955Z   NO_TEST_TIMEOUT: False
2025-12-04T12:28:08.6301050Z   NO_TD: False
2025-12-04T12:28:08.6301315Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:28:08.6301602Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 1
2025-12-04T12:28:08.6301729Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2025-12-04T12:28:08.6301844Z   TESTS_TO_INCLUDE: 
2025-12-04T12:28:08.6301938Z   DASHBOARD_TAG: 
2025-12-04T12:28:08.6302077Z   HUGGING_FACE_HUB_TOKEN: ***
2025-12-04T12:28:08.6302183Z ##[endgroup]
2025-12-04T12:28:08.6318432Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2025-12-04T12:28:08.6318677Z + [[ linux-noble-rocm-py3.12-mi300 == *onnx* ]]
2025-12-04T12:28:08.6318827Z + TEST_COMMAND=.ci/pytorch/test.sh
2025-12-04T12:28:08.6326693Z +++ nproc --ignore=2
2025-12-04T12:28:08.6336531Z ++ docker run --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/home/runner/_work/_temp/github_env_19922798714 --ulimit stack=10485760:83886080 --ulimit core=0 --env-file=/tmp/github_env_19922798714 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-noble-rocm-n-py3-f0cd68561080d537ef3d3d6f81b25a6416ad600a
2025-12-04T12:28:08.8250129Z + container_name=4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T12:28:08.8250413Z + echo CONTAINER_NAME=4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T12:28:08.8250814Z + docker exec -t 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh'
2025-12-04T12:28:12.3109978Z Processing ./dist/torch-2.10.0a0+gitffd9b0f-cp312-cp312-linux_x86_64.whl
2025-12-04T12:28:12.8297747Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+gitffd9b0f) (3.18.0)
2025-12-04T12:28:12.8298807Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+gitffd9b0f) (4.12.2)
2025-12-04T12:28:12.8299768Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+gitffd9b0f) (78.1.1)
2025-12-04T12:28:12.8302885Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+gitffd9b0f) (1.13.3)
2025-12-04T12:28:12.8303781Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+gitffd9b0f) (2.8.8)
2025-12-04T12:28:12.8304645Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+gitffd9b0f) (3.1.6)
2025-12-04T12:28:12.8305520Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from torch==2.10.0a0+gitffd9b0f) (2025.10.0)
2025-12-04T12:28:12.8351499Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from sympy>=1.13.3->torch==2.10.0a0+gitffd9b0f) (1.3.0)
2025-12-04T12:28:12.8370805Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.12/lib/python3.12/site-packages (from jinja2->torch==2.10.0a0+gitffd9b0f) (3.0.3)
2025-12-04T12:28:12.9465497Z Installing collected packages: torch
2025-12-04T12:28:18.4904868Z Successfully installed torch-2.10.0a0+gitffd9b0f
2025-12-04T12:28:18.5358794Z + export TERM=vt100
2025-12-04T12:28:18.5358955Z + TERM=vt100
2025-12-04T12:28:18.5363636Z ++ dirname .ci/pytorch/test.sh
2025-12-04T12:28:18.5373198Z + source .ci/pytorch/common.sh
2025-12-04T12:28:18.5376912Z +++ dirname .ci/pytorch/common.sh
2025-12-04T12:28:18.5386808Z ++ source .ci/pytorch/common_utils.sh
2025-12-04T12:28:18.5389116Z +++ declare -f -t trap_add
2025-12-04T12:28:18.5395293Z ++ set -ex -o pipefail
2025-12-04T12:28:18.5395546Z ++ [[ linux-noble-rocm-py3.12-mi300 == *rocm* ]]
2025-12-04T12:28:18.5395753Z ++ unset HIP_PLATFORM
2025-12-04T12:28:18.5395943Z ++ export PYTORCH_TEST_WITH_ROCM=1
2025-12-04T12:28:18.5396132Z ++ PYTORCH_TEST_WITH_ROCM=1
2025-12-04T12:28:18.5396306Z ++ BUILD_TEST_LIBTORCH=0
2025-12-04T12:28:18.5401850Z ++ dirname .ci/pytorch/test.sh
2025-12-04T12:28:18.5412011Z + source .ci/pytorch/common-build.sh
2025-12-04T12:28:18.5414344Z ++ [[ linux-noble-rocm-py3.12-mi300 != *win-* ]]
2025-12-04T12:28:18.5423638Z ++++ dirname .ci/pytorch/common-build.sh
2025-12-04T12:28:18.5435358Z +++ cd .ci/pytorch
2025-12-04T12:28:18.5435686Z +++ pwd -P
2025-12-04T12:28:18.5437493Z ++ script_dir=/var/lib/jenkins/pytorch/.ci/pytorch
2025-12-04T12:28:18.5437880Z ++ [[ linux-noble-rocm-py3.12-mi300 == *-pch* ]]
2025-12-04T12:28:18.5438766Z ++ which sccache
2025-12-04T12:28:18.5449754Z ++ [[ -z '' ]]
2025-12-04T12:28:18.5449932Z ++ unset SCCACHE_BUCKET
2025-12-04T12:28:18.5450107Z ++ unset SCCACHE_REGION
2025-12-04T12:28:18.5450279Z ++ sccache --stop-server
2025-12-04T12:28:18.5467942Z ++ true
2025-12-04T12:28:18.5469177Z ++ rm -f /var/lib/jenkins/sccache_error.log
2025-12-04T12:28:18.5477756Z ++ trap_add sccache_epilogue EXIT
2025-12-04T12:28:18.5478011Z ++ trap_add_cmd=sccache_epilogue
2025-12-04T12:28:18.5478245Z ++ shift
2025-12-04T12:28:18.5478413Z ++ for trap_add_name in "$@"
2025-12-04T12:28:18.5486880Z ++++ trap -p EXIT
2025-12-04T12:28:18.5488993Z +++ eval 'extract_trap_cmd '
2025-12-04T12:28:18.5489195Z ++++ extract_trap_cmd
2025-12-04T12:28:18.5489356Z ++++ printf '%s\n' ''
2025-12-04T12:28:18.5489530Z +++ printf '%s\n' sccache_epilogue
2025-12-04T12:28:18.5491700Z ++ trap -- '
2025-12-04T12:28:18.5492033Z sccache_epilogue' EXIT
2025-12-04T12:28:18.5492224Z ++ [[ -n '' ]]
2025-12-04T12:28:18.5492401Z ++ [[ linux-noble-rocm-py3.12-mi300 == *rocm* ]]
2025-12-04T12:28:18.5492657Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-12-04T12:28:18.5492874Z ++ SCCACHE_IDLE_TIMEOUT=0
2025-12-04T12:28:18.5493046Z ++ sccache --start-server
2025-12-04T12:28:18.5507170Z sccache: Starting the server...
2025-12-04T12:28:18.5708205Z sccache: Listening on address 127.0.0.1:4226
2025-12-04T12:28:18.5718631Z ++ sccache --zero-stats
2025-12-04T12:28:18.5737398Z Statistics zeroed.
2025-12-04T12:28:18.5741895Z ++ which ccache
2025-12-04T12:28:18.5750717Z + [[ linux-noble-rocm-py3.12-mi300 != *rocm* ]]
2025-12-04T12:28:18.5751727Z + [[ linux-noble-rocm-py3.12-mi300 == *cuda* ]]
2025-12-04T12:28:18.5751967Z + echo 'Environment variables:'
2025-12-04T12:28:18.5752110Z Environment variables:
2025-12-04T12:28:18.5752227Z + env
2025-12-04T12:28:18.5759691Z GITHUB_WORKSPACE=/home/runner/_work/pytorch/pytorch
2025-12-04T12:28:18.5759910Z CONTINUE_THROUGH_ERROR=True
2025-12-04T12:28:18.5760063Z BUILD_ENVIRONMENT=linux-noble-rocm-py3.12-mi300
2025-12-04T12:28:18.5760278Z HOSTNAME=linux.rocm.gpu.gfx942.4.b-bphpw-runner-bfc27
2025-12-04T12:28:18.5760534Z GITHUB_PATH=/home/runner/_work/_temp/_runner_file_commands/add_path_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5760746Z GITHUB_ACTION=__run_2
2025-12-04T12:28:18.5760866Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1
2025-12-04T12:28:18.5760994Z GITHUB_RUN_NUMBER=1861
2025-12-04T12:28:18.5761105Z TEST_CONFIG=distributed
2025-12-04T12:28:18.5761263Z RUNNER_NAME=linux.rocm.gpu.gfx942.4.b-bphpw-runner-bfc27
2025-12-04T12:28:18.5761421Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-12-04T12:28:18.5761547Z AWS_DEFAULT_REGION=us-east-1
2025-12-04T12:28:18.5761685Z RUNNER_ARTIFACT_DIR=/home/runner/_work/_temp/artifacts
2025-12-04T12:28:18.5761831Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2025-12-04T12:28:18.5761954Z GITHUB_REF_TYPE=branch
2025-12-04T12:28:18.5762080Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:18.5762349Z HUGGING_FACE_HUB_TOKEN=***
2025-12-04T12:28:18.5762775Z ***
2025-12-04T12:28:18.5762873Z GITHUB_REPOSITORY_ID=65600975
2025-12-04T12:28:18.5762991Z GITHUB_ACTIONS=true
2025-12-04T12:28:18.5763114Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:18.5763277Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:18.5763509Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic-rocm-mi300.yml@refs/heads/main
2025-12-04T12:28:18.5763714Z UCC_HOME=/usr
2025-12-04T12:28:18.5763816Z RUNNER_ENVIRONMENT=self-hosted
2025-12-04T12:28:18.5764355Z VERBOSE_TEST_LOGS=False
2025-12-04T12:28:18.5764467Z GITHUB_REF=refs/heads/main
2025-12-04T12:28:18.5764576Z RUNNER_OS=Linux
2025-12-04T12:28:18.5764672Z SHARD_NUMBER=3
2025-12-04T12:28:18.5764773Z GITHUB_REF_PROTECTED=true
2025-12-04T12:28:18.5764888Z RUNNER_MANUALLY_TRAP_SIG=1
2025-12-04T12:28:18.5764997Z HOME=/var/lib/jenkins
2025-12-04T12:28:18.5765122Z GITHUB_API_URL=https://api.github.com
2025-12-04T12:28:18.5765293Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-12-04T12:28:18.5765431Z RUNNER_DOCS_DIR=/home/runner/_work/_temp/docs
2025-12-04T12:28:18.5765679Z LANG=C.UTF-8
2025-12-04T12:28:18.5765796Z UCX_COMMIT=29831d319e6be55cb8c768ca61de335c934ca39e
2025-12-04T12:28:18.5765940Z PYTORCH_TEST_WITH_ROCM=1
2025-12-04T12:28:18.5766102Z RUNNER_TRACKING_ID=github_379f4657-ab94-499a-b4bb-a9f5748d5cb2
2025-12-04T12:28:18.5766256Z RUNNER_ARCH=X64
2025-12-04T12:28:18.5766360Z RUNNER_TEMP=/home/runner/_work/_temp
2025-12-04T12:28:18.5766483Z NUM_TEST_SHARDS=3
2025-12-04T12:28:18.5766580Z UCX_HOME=/usr
2025-12-04T12:28:18.5766833Z GITHUB_STATE=/home/runner/_work/_temp/_runner_file_commands/save_state_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5767190Z JOB_NAME=linux-noble-rocm-py3.12-mi300 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4.b, module:rocm, oncall:distributed, mem_leak_check)
2025-12-04T12:28:18.5767450Z MAGMA_HOME=/opt/rocm/magma
2025-12-04T12:28:18.5767649Z GITHUB_ENV=/home/runner/_work/_temp/_runner_file_commands/set_env_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5767906Z GITHUB_EVENT_PATH=/home/runner/_work/_temp/_github_workflow/event.json
2025-12-04T12:28:18.5768074Z GITHUB_EVENT_NAME=schedule
2025-12-04T12:28:18.5768264Z GITHUB_ACTIONS_RUNNER_EXTRA_USER_AGENT=actions-runner-controller/0.12.1
2025-12-04T12:28:18.5768430Z DASHBOARD_TAG=
2025-12-04T12:28:18.5768524Z GITHUB_RUN_ID=19922798714
2025-12-04T12:28:18.5768734Z GITHUB_STEP_SUMMARY=/home/runner/_work/_temp/_runner_file_commands/step_summary_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5768962Z GITHUB_ACTOR=pytorchmergebot
2025-12-04T12:28:18.5769071Z PR_NUMBER=
2025-12-04T12:28:18.5769161Z GITHUB_RUN_ATTEMPT=1
2025-12-04T12:28:18.5769266Z ANACONDA_PYTHON_VERSION=3.12
2025-12-04T12:28:18.5769398Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-12-04T12:28:18.5769530Z TERM=vt100
2025-12-04T12:28:18.5769618Z INSTALLED_VISION=yes
2025-12-04T12:28:18.5769717Z BRANCH=main
2025-12-04T12:28:18.5769810Z OPENSSL_ROOT_DIR=/opt/openssl
2025-12-04T12:28:18.5769918Z TESTS_TO_INCLUDE=
2025-12-04T12:28:18.5770082Z GITHUB_ACTION_PATH=/home/runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-12-04T12:28:18.5770270Z GITHUB_SERVER_URL=https://github.com
2025-12-04T12:28:18.5770407Z PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100
2025-12-04T12:28:18.5770557Z UCC_COMMIT=9f4b242cbbd8b1462cbc732eb29316cdfa124b77
2025-12-04T12:28:18.5770689Z REENABLED_ISSUES=
2025-12-04T12:28:18.5770781Z SHLVL=1
2025-12-04T12:28:18.5770868Z MAX_JOBS=126
2025-12-04T12:28:18.5770997Z RUNNER_TEST_RESULTS_DIR=/home/runner/_work/_temp/test-results
2025-12-04T12:28:18.5771147Z GITHUB_ACTOR_ID=97764156
2025-12-04T12:28:18.5771261Z RUNNER_TOOL_CACHE=/home/runner/_work/_tool
2025-12-04T12:28:18.5771421Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:18.5771569Z GITHUB_REF_NAME=main
2025-12-04T12:28:18.5771668Z ROCM_PATH=/opt/rocm
2025-12-04T12:28:18.5771763Z GITHUB_JOB=test
2025-12-04T12:28:18.5771858Z NO_TEST_TIMEOUT=False
2025-12-04T12:28:18.5771968Z GITHUB_REPOSITORY=pytorch/pytorch
2025-12-04T12:28:18.5772088Z LC_ALL=C.UTF-8
2025-12-04T12:28:18.5772182Z GITHUB_RETENTION_DAYS=90
2025-12-04T12:28:18.5772299Z RUNNER_WORKSPACE=/home/runner/_work/pytorch
2025-12-04T12:28:18.5772424Z OPENSSL_DIR=/opt/openssl
2025-12-04T12:28:18.5772531Z GITHUB_ACTION_REPOSITORY=
2025-12-04T12:28:18.5772888Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-12-04T12:28:18.5773274Z GITHUB_BASE_REF=
2025-12-04T12:28:18.5773366Z CI=true
2025-12-04T12:28:18.5773458Z GITHUB_REPOSITORY_OWNER=pytorch
2025-12-04T12:28:18.5773569Z JOB_ID=57117547539
2025-12-04T12:28:18.5773662Z GITHUB_HEAD_REF=
2025-12-04T12:28:18.5773754Z GITHUB_ACTION_REF=
2025-12-04T12:28:18.5773850Z TEST_SHOWLOCALS=False
2025-12-04T12:28:18.5773961Z GITHUB_WORKFLOW=periodic-rocm-mi300
2025-12-04T12:28:18.5774085Z DEBIAN_FRONTEND=noninteractive
2025-12-04T12:28:18.5774290Z GITHUB_OUTPUT=/home/runner/_work/_temp/_runner_file_commands/set_output_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5774530Z NO_TD=False
2025-12-04T12:28:18.5774621Z OLDPWD=/var/lib/jenkins
2025-12-04T12:28:18.5774720Z _=/usr/bin/env
2025-12-04T12:28:18.5774862Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2025-12-04T12:28:18.5837231Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch
2025-12-04T12:28:18.5837478Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/bin
2025-12-04T12:28:18.5837694Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/lib
2025-12-04T12:28:18.5837910Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/test
2025-12-04T12:28:18.5838073Z + BUILD_DIR=build
2025-12-04T12:28:18.5838174Z + BUILD_RENAMED_DIR=build_renamed
2025-12-04T12:28:18.5838291Z + BUILD_BIN_DIR=build/bin
2025-12-04T12:28:18.5838401Z + SHARD_NUMBER=3
2025-12-04T12:28:18.5838494Z + NUM_TEST_SHARDS=3
2025-12-04T12:28:18.5838606Z + export TORCH_SERIALIZATION_DEBUG=1
2025-12-04T12:28:18.5838730Z + TORCH_SERIALIZATION_DEBUG=1
2025-12-04T12:28:18.5838841Z + export VALGRIND=ON
2025-12-04T12:28:18.5838939Z + VALGRIND=ON
2025-12-04T12:28:18.5839050Z + [[ linux-noble-rocm-py3.12-mi300 == *clang9* ]]
2025-12-04T12:28:18.5839198Z + [[ linux-noble-rocm-py3.12-mi300 == *xpu* ]]
2025-12-04T12:28:18.5839321Z + detect_cuda_arch
2025-12-04T12:28:18.5839430Z + [[ linux-noble-rocm-py3.12-mi300 == *cuda* ]]
2025-12-04T12:28:18.5839577Z + [[ linux-noble-rocm-py3.12-mi300 == *s390x* ]]
2025-12-04T12:28:18.5839704Z + [[ 0 == \1 ]]
2025-12-04T12:28:18.5839794Z + [[ True == \1 ]]
2025-12-04T12:28:18.5839902Z + [[ linux-noble-rocm-py3.12-mi300 != *bazel* ]]
2025-12-04T12:28:18.5841826Z ++ realpath build/custom_test_artifacts
2025-12-04T12:28:18.5850682Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts
2025-12-04T12:28:18.5851079Z + [[ -n '' ]]
2025-12-04T12:28:18.5851271Z + echo 'Environment variables'
2025-12-04T12:28:18.5851511Z Environment variables
2025-12-04T12:28:18.5851689Z + env
2025-12-04T12:28:18.5856717Z GITHUB_WORKSPACE=/home/runner/_work/pytorch/pytorch
2025-12-04T12:28:18.5857049Z CONTINUE_THROUGH_ERROR=True
2025-12-04T12:28:18.5857288Z BUILD_ENVIRONMENT=linux-noble-rocm-py3.12-mi300
2025-12-04T12:28:18.5857592Z HOSTNAME=linux.rocm.gpu.gfx942.4.b-bphpw-runner-bfc27
2025-12-04T12:28:18.5858024Z GITHUB_PATH=/home/runner/_work/_temp/_runner_file_commands/add_path_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5858396Z GITHUB_ACTION=__run_2
2025-12-04T12:28:18.5858590Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1
2025-12-04T12:28:18.5858799Z GITHUB_RUN_NUMBER=1861
2025-12-04T12:28:18.5858983Z TEST_CONFIG=distributed
2025-12-04T12:28:18.5859230Z RUNNER_NAME=linux.rocm.gpu.gfx942.4.b-bphpw-runner-bfc27
2025-12-04T12:28:18.5859516Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-12-04T12:28:18.5859748Z AWS_DEFAULT_REGION=us-east-1
2025-12-04T12:28:18.5859975Z RUNNER_ARTIFACT_DIR=/home/runner/_work/_temp/artifacts
2025-12-04T12:28:18.5860228Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2025-12-04T12:28:18.5860432Z GITHUB_REF_TYPE=branch
2025-12-04T12:28:18.5860633Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:18.5861059Z HUGGING_FACE_HUB_TOKEN=***
2025-12-04T12:28:18.5861318Z ***
2025-12-04T12:28:18.5861477Z GITHUB_REPOSITORY_ID=65600975
2025-12-04T12:28:18.5861667Z GITHUB_ACTIONS=true
2025-12-04T12:28:18.5861853Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:18.5862352Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:18.5862715Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic-rocm-mi300.yml@refs/heads/main
2025-12-04T12:28:18.5863044Z UCC_HOME=/usr
2025-12-04T12:28:18.5863209Z TORCH_SERIALIZATION_DEBUG=1
2025-12-04T12:28:18.5863405Z RUNNER_ENVIRONMENT=self-hosted
2025-12-04T12:28:18.5863585Z VERBOSE_TEST_LOGS=False
2025-12-04T12:28:18.5863765Z GITHUB_REF=refs/heads/main
2025-12-04T12:28:18.5863935Z RUNNER_OS=Linux
2025-12-04T12:28:18.5864086Z SHARD_NUMBER=3
2025-12-04T12:28:18.5864433Z GITHUB_REF_PROTECTED=true
2025-12-04T12:28:18.5864615Z RUNNER_MANUALLY_TRAP_SIG=1
2025-12-04T12:28:18.5864785Z HOME=/var/lib/jenkins
2025-12-04T12:28:18.5864979Z GITHUB_API_URL=https://api.github.com
2025-12-04T12:28:18.5865199Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-12-04T12:28:18.5865412Z RUNNER_DOCS_DIR=/home/runner/_work/_temp/docs
2025-12-04T12:28:18.5865614Z LANG=C.UTF-8
2025-12-04T12:28:18.5865802Z UCX_COMMIT=29831d319e6be55cb8c768ca61de335c934ca39e
2025-12-04T12:28:18.5866027Z PYTORCH_TEST_WITH_ROCM=1
2025-12-04T12:28:18.5866268Z RUNNER_TRACKING_ID=github_379f4657-ab94-499a-b4bb-a9f5748d5cb2
2025-12-04T12:28:18.5866510Z RUNNER_ARCH=X64
2025-12-04T12:28:18.5866677Z RUNNER_TEMP=/home/runner/_work/_temp
2025-12-04T12:28:18.5866918Z NUM_TEST_SHARDS=3
2025-12-04T12:28:18.5867072Z UCX_HOME=/usr
2025-12-04T12:28:18.5867378Z GITHUB_STATE=/home/runner/_work/_temp/_runner_file_commands/save_state_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5867961Z JOB_NAME=linux-noble-rocm-py3.12-mi300 / test (distributed, 3, 3, linux.rocm.gpu.gfx942.4.b, module:rocm, oncall:distributed, mem_leak_check)
2025-12-04T12:28:18.5868376Z MAGMA_HOME=/opt/rocm/magma
2025-12-04T12:28:18.5868688Z GITHUB_ENV=/home/runner/_work/_temp/_runner_file_commands/set_env_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5869093Z GITHUB_EVENT_PATH=/home/runner/_work/_temp/_github_workflow/event.json
2025-12-04T12:28:18.5869354Z GITHUB_EVENT_NAME=schedule
2025-12-04T12:28:18.5869580Z GITHUB_ACTIONS_RUNNER_EXTRA_USER_AGENT=actions-runner-controller/0.12.1
2025-12-04T12:28:18.5869788Z DASHBOARD_TAG=
2025-12-04T12:28:18.5869912Z GITHUB_RUN_ID=19922798714
2025-12-04T12:28:18.5870171Z GITHUB_STEP_SUMMARY=/home/runner/_work/_temp/_runner_file_commands/step_summary_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5870452Z GITHUB_ACTOR=pytorchmergebot
2025-12-04T12:28:18.5870591Z PR_NUMBER=
2025-12-04T12:28:18.5870707Z GITHUB_RUN_ATTEMPT=1
2025-12-04T12:28:18.5870836Z VALGRIND=ON
2025-12-04T12:28:18.5870955Z ANACONDA_PYTHON_VERSION=3.12
2025-12-04T12:28:18.5871127Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-12-04T12:28:18.5871312Z TERM=vt100
2025-12-04T12:28:18.5871427Z INSTALLED_VISION=yes
2025-12-04T12:28:18.5871548Z BRANCH=main
2025-12-04T12:28:18.5871661Z OPENSSL_ROOT_DIR=/opt/openssl
2025-12-04T12:28:18.5871798Z TESTS_TO_INCLUDE=
2025-12-04T12:28:18.5871983Z GITHUB_ACTION_PATH=/home/runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-12-04T12:28:18.5872223Z GITHUB_SERVER_URL=https://github.com
2025-12-04T12:28:18.5872389Z PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100
2025-12-04T12:28:18.5872578Z UCC_COMMIT=9f4b242cbbd8b1462cbc732eb29316cdfa124b77
2025-12-04T12:28:18.5872738Z REENABLED_ISSUES=
2025-12-04T12:28:18.5872850Z SHLVL=1
2025-12-04T12:28:18.5872954Z MAX_JOBS=126
2025-12-04T12:28:18.5873111Z RUNNER_TEST_RESULTS_DIR=/home/runner/_work/_temp/test-results
2025-12-04T12:28:18.5873298Z GITHUB_ACTOR_ID=97764156
2025-12-04T12:28:18.5873439Z RUNNER_TOOL_CACHE=/home/runner/_work/_tool
2025-12-04T12:28:18.5873631Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32
2025-12-04T12:28:18.5873810Z GITHUB_REF_NAME=main
2025-12-04T12:28:18.5873929Z ROCM_PATH=/opt/rocm
2025-12-04T12:28:18.5874054Z GITHUB_JOB=test
2025-12-04T12:28:18.5874166Z NO_TEST_TIMEOUT=False
2025-12-04T12:28:18.5874295Z GITHUB_REPOSITORY=pytorch/pytorch
2025-12-04T12:28:18.5874433Z LC_ALL=C.UTF-8
2025-12-04T12:28:18.5874548Z GITHUB_RETENTION_DAYS=90
2025-12-04T12:28:18.5874739Z RUNNER_WORKSPACE=/home/runner/_work/pytorch
2025-12-04T12:28:18.5874893Z OPENSSL_DIR=/opt/openssl
2025-12-04T12:28:18.5875029Z GITHUB_ACTION_REPOSITORY=
2025-12-04T12:28:18.5875462Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-12-04T12:28:18.5875901Z GITHUB_BASE_REF=
2025-12-04T12:28:18.5876010Z CI=true
2025-12-04T12:28:18.5876119Z GITHUB_REPOSITORY_OWNER=pytorch
2025-12-04T12:28:18.5876306Z JOB_ID=57117547539
2025-12-04T12:28:18.5876417Z GITHUB_HEAD_REF=
2025-12-04T12:28:18.5876528Z GITHUB_ACTION_REF=
2025-12-04T12:28:18.5876644Z TEST_SHOWLOCALS=False
2025-12-04T12:28:18.5885683Z GITHUB_WORKFLOW=periodic-rocm-mi300
2025-12-04T12:28:18.5885839Z DEBIAN_FRONTEND=noninteractive
2025-12-04T12:28:18.5886054Z GITHUB_OUTPUT=/home/runner/_work/_temp/_runner_file_commands/set_output_5a58595f-fff6-4164-89fd-d74db470f648
2025-12-04T12:28:18.5886274Z NO_TD=False
2025-12-04T12:28:18.5886374Z OLDPWD=/var/lib/jenkins
2025-12-04T12:28:18.5886478Z _=/usr/bin/env
2025-12-04T12:28:18.5886577Z + echo 'Testing pytorch'
2025-12-04T12:28:18.5886682Z Testing pytorch
2025-12-04T12:28:18.5886834Z + export LANG=C.UTF-8
2025-12-04T12:28:18.5886932Z + LANG=C.UTF-8
2025-12-04T12:28:18.5887024Z + PR_NUMBER=
2025-12-04T12:28:18.5887127Z + [[ distributed == \d\e\f\a\u\l\t ]]
2025-12-04T12:28:18.5887260Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2025-12-04T12:28:18.5887404Z + [[ linux-noble-rocm-py3.12-mi300 == *rocm* ]]
2025-12-04T12:28:18.5887554Z + export HIP_VISIBLE_DEVICES=0,1,2,3
2025-12-04T12:28:18.5887682Z + HIP_VISIBLE_DEVICES=0,1,2,3
2025-12-04T12:28:18.5887802Z + [[ distributed == \s\l\o\w ]]
2025-12-04T12:28:18.5887948Z + [[ linux-noble-rocm-py3.12-mi300 == *slow-gradcheck* ]]
2025-12-04T12:28:18.5888109Z + [[ linux-noble-rocm-py3.12-mi300 == *cuda* ]]
2025-12-04T12:28:18.5888253Z + [[ linux-noble-rocm-py3.12-mi300 == *rocm* ]]
2025-12-04T12:28:18.5888398Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-12-04T12:28:18.5888537Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-12-04T12:28:18.5888667Z + [[ distributed == *crossref* ]]
2025-12-04T12:28:18.5888800Z + [[ linux-noble-rocm-py3.12-mi300 == *rocm* ]]
2025-12-04T12:28:18.5888929Z + export VALGRIND=OFF
2025-12-04T12:28:18.5889030Z + VALGRIND=OFF
2025-12-04T12:28:18.5889121Z + rocminfo
2025-12-04T12:28:18.5977421Z ROCk module version 6.12.12 is loaded
2025-12-04T12:28:18.6692211Z =====================    
2025-12-04T12:28:18.6692491Z HSA System Attributes    
2025-12-04T12:28:18.6692671Z =====================    
2025-12-04T12:28:18.6692854Z Runtime Version:         1.18
2025-12-04T12:28:18.6693044Z Runtime Ext Version:     1.14
2025-12-04T12:28:18.6693265Z System Timestamp Freq.:  1000.000000MHz
2025-12-04T12:28:18.6693588Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-12-04T12:28:18.6693931Z Machine Model:           LARGE                              
2025-12-04T12:28:18.6694229Z System Endianness:       LITTLE                             
2025-12-04T12:28:18.6694467Z Mwaitx:                  DISABLED
2025-12-04T12:28:18.6694660Z XNACK enabled:           NO
2025-12-04T12:28:18.6694857Z DMAbuf Support:          YES
2025-12-04T12:28:18.6695044Z VMM Support:             YES
2025-12-04T12:28:18.6695166Z 
2025-12-04T12:28:18.6695239Z ==========               
2025-12-04T12:28:18.6695418Z HSA Agents               
2025-12-04T12:28:18.6695588Z ==========               
2025-12-04T12:28:18.6695757Z *******                  
2025-12-04T12:28:18.6695925Z Agent 1                  
2025-12-04T12:28:18.6696095Z *******                  
2025-12-04T12:28:18.6696335Z   Name:                    AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:28:18.6696639Z   Uuid:                    CPU-XX                             
2025-12-04T12:28:18.6697018Z   Marketing Name:          AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:28:18.6697766Z   Vendor Name:             CPU                                
2025-12-04T12:28:18.6698078Z   Feature:                 None specified                     
2025-12-04T12:28:18.6698351Z   Profile:                 FULL_PROFILE                       
2025-12-04T12:28:18.6698666Z   Float Round Mode:        NEAR                               
2025-12-04T12:28:18.6698980Z   Max Queue Number:        0(0x0)                             
2025-12-04T12:28:18.6699308Z   Queue Min Size:          0(0x0)                             
2025-12-04T12:28:18.6699562Z   Queue Max Size:          0(0x0)                             
2025-12-04T12:28:18.6699969Z   Queue Type:              MULTI                              
2025-12-04T12:28:18.6700211Z   Node:                    0                                  
2025-12-04T12:28:18.6700418Z   Device Type:             CPU                                
2025-12-04T12:28:18.6700617Z   Cache Info:              
2025-12-04T12:28:18.6700794Z     L1:                      49152(0xc000) KB                   
2025-12-04T12:28:18.6700998Z   Chip ID:                 0(0x0)                             
2025-12-04T12:28:18.6701211Z   ASIC Revision:           0(0x0)                             
2025-12-04T12:28:18.6701432Z   Cacheline Size:          64(0x40)                           
2025-12-04T12:28:18.6701657Z   Max Clock Freq. (MHz):   3300                               
2025-12-04T12:28:18.6701869Z   BDFID:                   0                                  
2025-12-04T12:28:18.6702084Z   Internal Node ID:        0                                  
2025-12-04T12:28:18.6702307Z   Compute Unit:            64                                 
2025-12-04T12:28:18.6702527Z   SIMDs per CU:            0                                  
2025-12-04T12:28:18.6702760Z   Shader Engines:          0                                  
2025-12-04T12:28:18.6702982Z   Shader Arrs. per Eng.:   0                                  
2025-12-04T12:28:18.6703219Z   WatchPts on Addr. Ranges:1                                  
2025-12-04T12:28:18.6703431Z   Memory Properties:       
2025-12-04T12:28:18.6703722Z   Features:                None
2025-12-04T12:28:18.6703894Z   Pool Info:               
2025-12-04T12:28:18.6704138Z     Pool 1                   
2025-12-04T12:28:18.6704342Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:28:18.6704656Z       Size:                    1584734448(0x5e7520f0) KB          
2025-12-04T12:28:18.6704891Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6705273Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6705535Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:28:18.6705784Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6706092Z       Accessible by all:       TRUE                               
2025-12-04T12:28:18.6706296Z     Pool 2                   
2025-12-04T12:28:18.6706500Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:28:18.6706838Z       Size:                    1584734448(0x5e7520f0) KB          
2025-12-04T12:28:18.6707062Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6707340Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6707590Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:28:18.6707858Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6708121Z       Accessible by all:       TRUE                               
2025-12-04T12:28:18.6708330Z     Pool 3                   
2025-12-04T12:28:18.6708538Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-12-04T12:28:18.6708754Z       Size:                    1584734448(0x5e7520f0) KB          
2025-12-04T12:28:18.6708967Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6709251Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6709706Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:28:18.6709974Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6710153Z       Accessible by all:       TRUE                               
2025-12-04T12:28:18.6710311Z     Pool 4                   
2025-12-04T12:28:18.6710460Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:28:18.6710739Z       Size:                    1584734448(0x5e7520f0) KB          
2025-12-04T12:28:18.6710954Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6711131Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6711315Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:28:18.6711502Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6711682Z       Accessible by all:       TRUE                               
2025-12-04T12:28:18.6711846Z   ISA Info:                
2025-12-04T12:28:18.6711967Z *******                  
2025-12-04T12:28:18.6712078Z Agent 2                  
2025-12-04T12:28:18.6712191Z *******                  
2025-12-04T12:28:18.6712327Z   Name:                    AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:28:18.6712494Z   Uuid:                    CPU-XX                             
2025-12-04T12:28:18.6712669Z   Marketing Name:          AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:28:18.6712853Z   Vendor Name:             CPU                                
2025-12-04T12:28:18.6713030Z   Feature:                 None specified                     
2025-12-04T12:28:18.6713206Z   Profile:                 FULL_PROFILE                       
2025-12-04T12:28:18.6713378Z   Float Round Mode:        NEAR                               
2025-12-04T12:28:18.6713553Z   Max Queue Number:        0(0x0)                             
2025-12-04T12:28:18.6713729Z   Queue Min Size:          0(0x0)                             
2025-12-04T12:28:18.6713900Z   Queue Max Size:          0(0x0)                             
2025-12-04T12:28:18.6714071Z   Queue Type:              MULTI                              
2025-12-04T12:28:18.6714232Z   Node:                    1                                  
2025-12-04T12:28:18.6714394Z   Device Type:             CPU                                
2025-12-04T12:28:18.6714548Z   Cache Info:              
2025-12-04T12:28:18.6714681Z     L1:                      49152(0xc000) KB                   
2025-12-04T12:28:18.6714840Z   Chip ID:                 0(0x0)                             
2025-12-04T12:28:18.6715005Z   ASIC Revision:           0(0x0)                             
2025-12-04T12:28:18.6715179Z   Cacheline Size:          64(0x40)                           
2025-12-04T12:28:18.6715356Z   Max Clock Freq. (MHz):   3300                               
2025-12-04T12:28:18.6715522Z   BDFID:                   0                                  
2025-12-04T12:28:18.6715693Z   Internal Node ID:        1                                  
2025-12-04T12:28:18.6715865Z   Compute Unit:            64                                 
2025-12-04T12:28:18.6716033Z   SIMDs per CU:            0                                  
2025-12-04T12:28:18.6716209Z   Shader Engines:          0                                  
2025-12-04T12:28:18.6716387Z   Shader Arrs. per Eng.:   0                                  
2025-12-04T12:28:18.6716574Z   WatchPts on Addr. Ranges:1                                  
2025-12-04T12:28:18.6716736Z   Memory Properties:       
2025-12-04T12:28:18.6716907Z   Features:                None
2025-12-04T12:28:18.6717030Z   Pool Info:               
2025-12-04T12:28:18.6717150Z     Pool 1                   
2025-12-04T12:28:18.6717299Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:28:18.6717477Z       Size:                    1585355632(0x5e7e9b70) KB          
2025-12-04T12:28:18.6717696Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6717890Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6718095Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:28:18.6718283Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6718466Z       Accessible by all:       TRUE                               
2025-12-04T12:28:18.6718636Z     Pool 2                   
2025-12-04T12:28:18.6718784Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:28:18.6718995Z       Size:                    1585355632(0x5e7e9b70) KB          
2025-12-04T12:28:18.6719235Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6719417Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6719610Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:28:18.6719830Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6720062Z       Accessible by all:       TRUE                               
2025-12-04T12:28:18.6720229Z     Pool 3                   
2025-12-04T12:28:18.6720393Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-12-04T12:28:18.6720559Z       Size:                    1585355632(0x5e7e9b70) KB          
2025-12-04T12:28:18.6720788Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6720981Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6721153Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:28:18.6721322Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6721596Z       Accessible by all:       TRUE                               
2025-12-04T12:28:18.6721813Z     Pool 4                   
2025-12-04T12:28:18.6721986Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:28:18.6722222Z       Size:                    1585355632(0x5e7e9b70) KB          
2025-12-04T12:28:18.6722378Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6722699Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6722891Z       Alloc Recommended Granule:4KB                                
2025-12-04T12:28:18.6723062Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6723229Z       Accessible by all:       TRUE                               
2025-12-04T12:28:18.6723415Z   ISA Info:                
2025-12-04T12:28:18.6723524Z *******                  
2025-12-04T12:28:18.6723695Z Agent 3                  
2025-12-04T12:28:18.6723801Z *******                  
2025-12-04T12:28:18.6723923Z   Name:                    gfx942                             
2025-12-04T12:28:18.6724074Z   Uuid:                    GPU-dd105c7c944d0f95               
2025-12-04T12:28:18.6724260Z   Marketing Name:          AMD Radeon Graphics                
2025-12-04T12:28:18.6724425Z   Vendor Name:             AMD                                
2025-12-04T12:28:18.6724586Z   Feature:                 KERNEL_DISPATCH                    
2025-12-04T12:28:18.6724821Z   Profile:                 BASE_PROFILE                       
2025-12-04T12:28:18.6724981Z   Float Round Mode:        NEAR                               
2025-12-04T12:28:18.6725144Z   Max Queue Number:        128(0x80)                          
2025-12-04T12:28:18.6725308Z   Queue Min Size:          64(0x40)                           
2025-12-04T12:28:18.6725468Z   Queue Max Size:          131072(0x20000)                    
2025-12-04T12:28:18.6725626Z   Queue Type:              MULTI                              
2025-12-04T12:28:18.6725774Z   Node:                    2                                  
2025-12-04T12:28:18.6726003Z   Device Type:             GPU                                
2025-12-04T12:28:18.6726141Z   Cache Info:              
2025-12-04T12:28:18.6726322Z     L1:                      32(0x20) KB                        
2025-12-04T12:28:18.6726526Z     L2:                      4096(0x1000) KB                    
2025-12-04T12:28:18.6726693Z     L3:                      262144(0x40000) KB                 
2025-12-04T12:28:18.6726904Z   Chip ID:                 29861(0x74a5)                      
2025-12-04T12:28:18.6727057Z   ASIC Revision:           1(0x1)                             
2025-12-04T12:28:18.6727221Z   Cacheline Size:          128(0x80)                          
2025-12-04T12:28:18.6727438Z   Max Clock Freq. (MHz):   2100                               
2025-12-04T12:28:18.6727644Z   BDFID:                   29952                              
2025-12-04T12:28:18.6727816Z   Internal Node ID:        2                                  
2025-12-04T12:28:18.6727976Z   Compute Unit:            304                                
2025-12-04T12:28:18.6728130Z   SIMDs per CU:            4                                  
2025-12-04T12:28:18.6728295Z   Shader Engines:          32                                 
2025-12-04T12:28:18.6728458Z   Shader Arrs. per Eng.:   1                                  
2025-12-04T12:28:18.6728627Z   WatchPts on Addr. Ranges:4                                  
2025-12-04T12:28:18.6728798Z   Coherent Host Access:    FALSE                              
2025-12-04T12:28:18.6728946Z   Memory Properties:       
2025-12-04T12:28:18.6729072Z   Features:                KERNEL_DISPATCH 
2025-12-04T12:28:18.6729223Z   Fast F16 Operation:      TRUE                               
2025-12-04T12:28:18.6729394Z   Wavefront Size:          64(0x40)                           
2025-12-04T12:28:18.6729630Z   Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6729795Z   Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6729962Z     x                        1024(0x400)                        
2025-12-04T12:28:18.6730134Z     y                        1024(0x400)                        
2025-12-04T12:28:18.6730342Z     z                        1024(0x400)                        
2025-12-04T12:28:18.6730602Z   Max Waves Per CU:        32(0x20)                           
2025-12-04T12:28:18.6730766Z   Max Work-item Per CU:    2048(0x800)                        
2025-12-04T12:28:18.6730932Z   Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6731093Z   Grid Max Size per Dimension:
2025-12-04T12:28:18.6731229Z     x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6731417Z     y                        65535(0xffff)                      
2025-12-04T12:28:18.6731549Z     z                        65535(0xffff)                      
2025-12-04T12:28:18.6731702Z   Max fbarriers/Workgrp:   32                                 
2025-12-04T12:28:18.6731953Z   Packet Processor uCode:: 185                                
2025-12-04T12:28:18.6732177Z   SDMA engine uCode::      24                                 
2025-12-04T12:28:18.6732342Z   IOMMU Support::          None                               
2025-12-04T12:28:18.6732536Z   Pool Info:               
2025-12-04T12:28:18.6732649Z     Pool 1                   
2025-12-04T12:28:18.6732791Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:28:18.6733021Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6733179Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6733350Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6733620Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6733791Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6733957Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6734136Z     Pool 2                   
2025-12-04T12:28:18.6734285Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:28:18.6734494Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6734654Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6734828Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6735075Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6735244Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6735417Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6735591Z     Pool 3                   
2025-12-04T12:28:18.6735797Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:28:18.6735980Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6736159Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6736324Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6736508Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6736774Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6736941Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6737088Z     Pool 4                   
2025-12-04T12:28:18.6737291Z       Segment:                 GROUP                              
2025-12-04T12:28:18.6737462Z       Size:                    64(0x40) KB                        
2025-12-04T12:28:18.6737682Z       Allocatable:             FALSE                              
2025-12-04T12:28:18.6737852Z       Alloc Granule:           0KB                                
2025-12-04T12:28:18.6738075Z       Alloc Recommended Granule:0KB                                
2025-12-04T12:28:18.6738342Z       Alloc Alignment:         0KB                                
2025-12-04T12:28:18.6738516Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6738665Z   ISA Info:                
2025-12-04T12:28:18.6738828Z     ISA 1                    
2025-12-04T12:28:18.6738967Z       Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
2025-12-04T12:28:18.6739139Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:28:18.6739324Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:28:18.6739490Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6739677Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6739870Z       Fast f16:                TRUE                               
2025-12-04T12:28:18.6740031Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6740186Z       Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6740323Z         x                        1024(0x400)                        
2025-12-04T12:28:18.6740529Z         y                        1024(0x400)                        
2025-12-04T12:28:18.6740696Z         z                        1024(0x400)                        
2025-12-04T12:28:18.6740907Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6741053Z       Grid Max Size per Dimension:
2025-12-04T12:28:18.6741181Z         x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6741371Z         y                        65535(0xffff)                      
2025-12-04T12:28:18.6741511Z         z                        65535(0xffff)                      
2025-12-04T12:28:18.6741678Z       FBarrier Max Size:       32                                 
2025-12-04T12:28:18.6741886Z     ISA 2                    
2025-12-04T12:28:18.6742083Z       Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
2025-12-04T12:28:18.6742265Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:28:18.6742436Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:28:18.6742712Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6742883Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6743047Z       Fast f16:                TRUE                               
2025-12-04T12:28:18.6743267Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6743419Z       Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6743552Z         x                        1024(0x400)                        
2025-12-04T12:28:18.6743791Z         y                        1024(0x400)                        
2025-12-04T12:28:18.6743959Z         z                        1024(0x400)                        
2025-12-04T12:28:18.6744174Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6744321Z       Grid Max Size per Dimension:
2025-12-04T12:28:18.6744533Z         x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6744676Z         y                        65535(0xffff)                      
2025-12-04T12:28:18.6744826Z         z                        65535(0xffff)                      
2025-12-04T12:28:18.6744982Z       FBarrier Max Size:       32                                 
2025-12-04T12:28:18.6745126Z *******                  
2025-12-04T12:28:18.6745303Z Agent 4                  
2025-12-04T12:28:18.6745457Z *******                  
2025-12-04T12:28:18.6745617Z   Name:                    gfx942                             
2025-12-04T12:28:18.6745777Z   Uuid:                    GPU-6f0efc92194cd6bf               
2025-12-04T12:28:18.6745940Z   Marketing Name:          AMD Radeon Graphics                
2025-12-04T12:28:18.6746118Z   Vendor Name:             AMD                                
2025-12-04T12:28:18.6746293Z   Feature:                 KERNEL_DISPATCH                    
2025-12-04T12:28:18.6746465Z   Profile:                 BASE_PROFILE                       
2025-12-04T12:28:18.6746667Z   Float Round Mode:        NEAR                               
2025-12-04T12:28:18.6746905Z   Max Queue Number:        128(0x80)                          
2025-12-04T12:28:18.6747143Z   Queue Min Size:          64(0x40)                           
2025-12-04T12:28:18.6747298Z   Queue Max Size:          131072(0x20000)                    
2025-12-04T12:28:18.6747452Z   Queue Type:              MULTI                              
2025-12-04T12:28:18.6747598Z   Node:                    3                                  
2025-12-04T12:28:18.6747745Z   Device Type:             GPU                                
2025-12-04T12:28:18.6747883Z   Cache Info:              
2025-12-04T12:28:18.6748001Z     L1:                      32(0x20) KB                        
2025-12-04T12:28:18.6748139Z     L2:                      4096(0x1000) KB                    
2025-12-04T12:28:18.6748277Z     L3:                      262144(0x40000) KB                 
2025-12-04T12:28:18.6748414Z   Chip ID:                 29861(0x74a5)                      
2025-12-04T12:28:18.6748665Z   ASIC Revision:           1(0x1)                             
2025-12-04T12:28:18.6748852Z   Cacheline Size:          128(0x80)                          
2025-12-04T12:28:18.6749049Z   Max Clock Freq. (MHz):   2100                               
2025-12-04T12:28:18.6749208Z   BDFID:                   1280                               
2025-12-04T12:28:18.6749369Z   Internal Node ID:        3                                  
2025-12-04T12:28:18.6749555Z   Compute Unit:            304                                
2025-12-04T12:28:18.6749725Z   SIMDs per CU:            4                                  
2025-12-04T12:28:18.6749920Z   Shader Engines:          32                                 
2025-12-04T12:28:18.6750087Z   Shader Arrs. per Eng.:   1                                  
2025-12-04T12:28:18.6750257Z   WatchPts on Addr. Ranges:4                                  
2025-12-04T12:28:18.6750467Z   Coherent Host Access:    FALSE                              
2025-12-04T12:28:18.6750676Z   Memory Properties:       
2025-12-04T12:28:18.6750836Z   Features:                KERNEL_DISPATCH 
2025-12-04T12:28:18.6751101Z   Fast F16 Operation:      TRUE                               
2025-12-04T12:28:18.6751273Z   Wavefront Size:          64(0x40)                           
2025-12-04T12:28:18.6751457Z   Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6751626Z   Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6751762Z     x                        1024(0x400)                        
2025-12-04T12:28:18.6751959Z     y                        1024(0x400)                        
2025-12-04T12:28:18.6752102Z     z                        1024(0x400)                        
2025-12-04T12:28:18.6752255Z   Max Waves Per CU:        32(0x20)                           
2025-12-04T12:28:18.6752601Z   Max Work-item Per CU:    2048(0x800)                        
2025-12-04T12:28:18.6752772Z   Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6752998Z   Grid Max Size per Dimension:
2025-12-04T12:28:18.6753126Z     x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6753267Z     y                        65535(0xffff)                      
2025-12-04T12:28:18.6753481Z     z                        65535(0xffff)                      
2025-12-04T12:28:18.6753638Z   Max fbarriers/Workgrp:   32                                 
2025-12-04T12:28:18.6753816Z   Packet Processor uCode:: 185                                
2025-12-04T12:28:18.6754084Z   SDMA engine uCode::      24                                 
2025-12-04T12:28:18.6754250Z   IOMMU Support::          None                               
2025-12-04T12:28:18.6754406Z   Pool Info:               
2025-12-04T12:28:18.6754530Z     Pool 1                   
2025-12-04T12:28:18.6754671Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:28:18.6754833Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6754997Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6755164Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6755337Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6755512Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6755681Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6755831Z     Pool 2                   
2025-12-04T12:28:18.6755975Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:28:18.6756134Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6756289Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6756455Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6756627Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6756867Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6757036Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6757180Z     Pool 3                   
2025-12-04T12:28:18.6757317Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:28:18.6757475Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6757633Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6757801Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6757975Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6758148Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6758316Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6758463Z     Pool 4                   
2025-12-04T12:28:18.6758642Z       Segment:                 GROUP                              
2025-12-04T12:28:18.6758796Z       Size:                    64(0x40) KB                        
2025-12-04T12:28:18.6758953Z       Allocatable:             FALSE                              
2025-12-04T12:28:18.6759118Z       Alloc Granule:           0KB                                
2025-12-04T12:28:18.6759292Z       Alloc Recommended Granule:0KB                                
2025-12-04T12:28:18.6759466Z       Alloc Alignment:         0KB                                
2025-12-04T12:28:18.6759673Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6759824Z   ISA Info:                
2025-12-04T12:28:18.6759939Z     ISA 1                    
2025-12-04T12:28:18.6760080Z       Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
2025-12-04T12:28:18.6760256Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:28:18.6760427Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:28:18.6760604Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6760778Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6760941Z       Fast f16:                TRUE                               
2025-12-04T12:28:18.6761108Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6761264Z       Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6761402Z         x                        1024(0x400)                        
2025-12-04T12:28:18.6761550Z         y                        1024(0x400)                        
2025-12-04T12:28:18.6761688Z         z                        1024(0x400)                        
2025-12-04T12:28:18.6761840Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6761988Z       Grid Max Size per Dimension:
2025-12-04T12:28:18.6762119Z         x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6762264Z         y                        65535(0xffff)                      
2025-12-04T12:28:18.6762401Z         z                        65535(0xffff)                      
2025-12-04T12:28:18.6762556Z       FBarrier Max Size:       32                                 
2025-12-04T12:28:18.6762703Z     ISA 2                    
2025-12-04T12:28:18.6762854Z       Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
2025-12-04T12:28:18.6763040Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:28:18.6763215Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:28:18.6763387Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6763562Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6763726Z       Fast f16:                TRUE                               
2025-12-04T12:28:18.6763893Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6764050Z       Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6764187Z         x                        1024(0x400)                        
2025-12-04T12:28:18.6764327Z         y                        1024(0x400)                        
2025-12-04T12:28:18.6764467Z         z                        1024(0x400)                        
2025-12-04T12:28:18.6764618Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6764768Z       Grid Max Size per Dimension:
2025-12-04T12:28:18.6764901Z         x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6765043Z         y                        65535(0xffff)                      
2025-12-04T12:28:18.6765183Z         z                        65535(0xffff)                      
2025-12-04T12:28:18.6765338Z       FBarrier Max Size:       32                                 
2025-12-04T12:28:18.6765484Z *******                  
2025-12-04T12:28:18.6765597Z Agent 5                  
2025-12-04T12:28:18.6765741Z *******                  
2025-12-04T12:28:18.6765867Z   Name:                    gfx942                             
2025-12-04T12:28:18.6766023Z   Uuid:                    GPU-f619d38b8c3dfd9b               
2025-12-04T12:28:18.6766190Z   Marketing Name:          AMD Radeon Graphics                
2025-12-04T12:28:18.6766356Z   Vendor Name:             AMD                                
2025-12-04T12:28:18.6766522Z   Feature:                 KERNEL_DISPATCH                    
2025-12-04T12:28:18.6766713Z   Profile:                 BASE_PROFILE                       
2025-12-04T12:28:18.6766921Z   Float Round Mode:        NEAR                               
2025-12-04T12:28:18.6767089Z   Max Queue Number:        128(0x80)                          
2025-12-04T12:28:18.6767253Z   Queue Min Size:          64(0x40)                           
2025-12-04T12:28:18.6767413Z   Queue Max Size:          131072(0x20000)                    
2025-12-04T12:28:18.6767579Z   Queue Type:              MULTI                              
2025-12-04T12:28:18.6767732Z   Node:                    4                                  
2025-12-04T12:28:18.6767887Z   Device Type:             GPU                                
2025-12-04T12:28:18.6768032Z   Cache Info:              
2025-12-04T12:28:18.6768158Z     L1:                      32(0x20) KB                        
2025-12-04T12:28:18.6768300Z     L2:                      4096(0x1000) KB                    
2025-12-04T12:28:18.6768444Z     L3:                      262144(0x40000) KB                 
2025-12-04T12:28:18.6768595Z   Chip ID:                 29861(0x74a5)                      
2025-12-04T12:28:18.6768758Z   ASIC Revision:           1(0x1)                             
2025-12-04T12:28:18.6768924Z   Cacheline Size:          128(0x80)                          
2025-12-04T12:28:18.6769087Z   Max Clock Freq. (MHz):   2100                               
2025-12-04T12:28:18.6769242Z   BDFID:                   25856                              
2025-12-04T12:28:18.6769402Z   Internal Node ID:        4                                  
2025-12-04T12:28:18.6769567Z   Compute Unit:            304                                
2025-12-04T12:28:18.6769727Z   SIMDs per CU:            4                                  
2025-12-04T12:28:18.6769890Z   Shader Engines:          32                                 
2025-12-04T12:28:18.6770056Z   Shader Arrs. per Eng.:   1                                  
2025-12-04T12:28:18.6770226Z   WatchPts on Addr. Ranges:4                                  
2025-12-04T12:28:18.6770401Z   Coherent Host Access:    FALSE                              
2025-12-04T12:28:18.6770549Z   Memory Properties:       
2025-12-04T12:28:18.6770675Z   Features:                KERNEL_DISPATCH 
2025-12-04T12:28:18.6770830Z   Fast F16 Operation:      TRUE                               
2025-12-04T12:28:18.6770993Z   Wavefront Size:          64(0x40)                           
2025-12-04T12:28:18.6771158Z   Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6771311Z   Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6771441Z     x                        1024(0x400)                        
2025-12-04T12:28:18.6771574Z     y                        1024(0x400)                        
2025-12-04T12:28:18.6771710Z     z                        1024(0x400)                        
2025-12-04T12:28:18.6771856Z   Max Waves Per CU:        32(0x20)                           
2025-12-04T12:28:18.6772017Z   Max Work-item Per CU:    2048(0x800)                        
2025-12-04T12:28:18.6772179Z   Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6772320Z   Grid Max Size per Dimension:
2025-12-04T12:28:18.6772442Z     x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6772577Z     y                        65535(0xffff)                      
2025-12-04T12:28:18.6772709Z     z                        65535(0xffff)                      
2025-12-04T12:28:18.6772951Z   Max fbarriers/Workgrp:   32                                 
2025-12-04T12:28:18.6773123Z   Packet Processor uCode:: 185                                
2025-12-04T12:28:18.6773290Z   SDMA engine uCode::      24                                 
2025-12-04T12:28:18.6773452Z   IOMMU Support::          None                               
2025-12-04T12:28:18.6773592Z   Pool Info:               
2025-12-04T12:28:18.6773702Z     Pool 1                   
2025-12-04T12:28:18.6773838Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:28:18.6774030Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6774185Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6774346Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6774515Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6774691Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6774860Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6775003Z     Pool 2                   
2025-12-04T12:28:18.6775139Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:28:18.6775295Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6775448Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6775609Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6775781Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6775949Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6776113Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6776261Z     Pool 3                   
2025-12-04T12:28:18.6776393Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:28:18.6776554Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6776711Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6776907Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6777076Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6777251Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6777417Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6777566Z     Pool 4                   
2025-12-04T12:28:18.6777697Z       Segment:                 GROUP                              
2025-12-04T12:28:18.6777846Z       Size:                    64(0x40) KB                        
2025-12-04T12:28:18.6778001Z       Allocatable:             FALSE                              
2025-12-04T12:28:18.6778165Z       Alloc Granule:           0KB                                
2025-12-04T12:28:18.6778340Z       Alloc Recommended Granule:0KB                                
2025-12-04T12:28:18.6778506Z       Alloc Alignment:         0KB                                
2025-12-04T12:28:18.6778669Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6778807Z   ISA Info:                
2025-12-04T12:28:18.6778914Z     ISA 1                    
2025-12-04T12:28:18.6779045Z       Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
2025-12-04T12:28:18.6779216Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:28:18.6779388Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:28:18.6779552Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6779718Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6779872Z       Fast f16:                TRUE                               
2025-12-04T12:28:18.6780026Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6780213Z       Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6780344Z         x                        1024(0x400)                        
2025-12-04T12:28:18.6780481Z         y                        1024(0x400)                        
2025-12-04T12:28:18.6780617Z         z                        1024(0x400)                        
2025-12-04T12:28:18.6780763Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6780904Z       Grid Max Size per Dimension:
2025-12-04T12:28:18.6781059Z         x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6781195Z         y                        65535(0xffff)                      
2025-12-04T12:28:18.6781329Z         z                        65535(0xffff)                      
2025-12-04T12:28:18.6781478Z       FBarrier Max Size:       32                                 
2025-12-04T12:28:18.6781627Z     ISA 2                    
2025-12-04T12:28:18.6781777Z       Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
2025-12-04T12:28:18.6781959Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:28:18.6782126Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:28:18.6782287Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6782455Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6782608Z       Fast f16:                TRUE                               
2025-12-04T12:28:18.6782766Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6782911Z       Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6783044Z         x                        1024(0x400)                        
2025-12-04T12:28:18.6783185Z         y                        1024(0x400)                        
2025-12-04T12:28:18.6783322Z         z                        1024(0x400)                        
2025-12-04T12:28:18.6783475Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6783622Z       Grid Max Size per Dimension:
2025-12-04T12:28:18.6783751Z         x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6783889Z         y                        65535(0xffff)                      
2025-12-04T12:28:18.6784030Z         z                        65535(0xffff)                      
2025-12-04T12:28:18.6784180Z       FBarrier Max Size:       32                                 
2025-12-04T12:28:18.6784323Z *******                  
2025-12-04T12:28:18.6784429Z Agent 6                  
2025-12-04T12:28:18.6784534Z *******                  
2025-12-04T12:28:18.6784655Z   Name:                    gfx942                             
2025-12-04T12:28:18.6784806Z   Uuid:                    GPU-8052c781c18bf05c               
2025-12-04T12:28:18.6784968Z   Marketing Name:          AMD Radeon Graphics                
2025-12-04T12:28:18.6785136Z   Vendor Name:             AMD                                
2025-12-04T12:28:18.6785293Z   Feature:                 KERNEL_DISPATCH                    
2025-12-04T12:28:18.6785449Z   Profile:                 BASE_PROFILE                       
2025-12-04T12:28:18.6785611Z   Float Round Mode:        NEAR                               
2025-12-04T12:28:18.6785774Z   Max Queue Number:        128(0x80)                          
2025-12-04T12:28:18.6785933Z   Queue Min Size:          64(0x40)                           
2025-12-04T12:28:18.6786088Z   Queue Max Size:          131072(0x20000)                    
2025-12-04T12:28:18.6786248Z   Queue Type:              MULTI                              
2025-12-04T12:28:18.6786402Z   Node:                    5                                  
2025-12-04T12:28:18.6786555Z   Device Type:             GPU                                
2025-12-04T12:28:18.6786698Z   Cache Info:              
2025-12-04T12:28:18.6786874Z     L1:                      32(0x20) KB                        
2025-12-04T12:28:18.6787070Z     L2:                      4096(0x1000) KB                    
2025-12-04T12:28:18.6787214Z     L3:                      262144(0x40000) KB                 
2025-12-04T12:28:18.6787355Z   Chip ID:                 29861(0x74a5)                      
2025-12-04T12:28:18.6787508Z   ASIC Revision:           1(0x1)                             
2025-12-04T12:28:18.6787674Z   Cacheline Size:          128(0x80)                          
2025-12-04T12:28:18.6787839Z   Max Clock Freq. (MHz):   2100                               
2025-12-04T12:28:18.6788030Z   BDFID:                   5376                               
2025-12-04T12:28:18.6788185Z   Internal Node ID:        5                                  
2025-12-04T12:28:18.6788346Z   Compute Unit:            304                                
2025-12-04T12:28:18.6788501Z   SIMDs per CU:            4                                  
2025-12-04T12:28:18.6788660Z   Shader Engines:          32                                 
2025-12-04T12:28:18.6788828Z   Shader Arrs. per Eng.:   1                                  
2025-12-04T12:28:18.6788989Z   WatchPts on Addr. Ranges:4                                  
2025-12-04T12:28:18.6789153Z   Coherent Host Access:    FALSE                              
2025-12-04T12:28:18.6789296Z   Memory Properties:       
2025-12-04T12:28:18.6789415Z   Features:                KERNEL_DISPATCH 
2025-12-04T12:28:18.6789561Z   Fast F16 Operation:      TRUE                               
2025-12-04T12:28:18.6789724Z   Wavefront Size:          64(0x40)                           
2025-12-04T12:28:18.6789891Z   Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6790036Z   Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6790160Z     x                        1024(0x400)                        
2025-12-04T12:28:18.6790289Z     y                        1024(0x400)                        
2025-12-04T12:28:18.6790418Z     z                        1024(0x400)                        
2025-12-04T12:28:18.6790565Z   Max Waves Per CU:        32(0x20)                           
2025-12-04T12:28:18.6790722Z   Max Work-item Per CU:    2048(0x800)                        
2025-12-04T12:28:18.6790879Z   Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6791018Z   Grid Max Size per Dimension:
2025-12-04T12:28:18.6791136Z     x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6791269Z     y                        65535(0xffff)                      
2025-12-04T12:28:18.6791404Z     z                        65535(0xffff)                      
2025-12-04T12:28:18.6791550Z   Max fbarriers/Workgrp:   32                                 
2025-12-04T12:28:18.6791724Z   Packet Processor uCode:: 185                                
2025-12-04T12:28:18.6791892Z   SDMA engine uCode::      24                                 
2025-12-04T12:28:18.6792053Z   IOMMU Support::          None                               
2025-12-04T12:28:18.6792194Z   Pool Info:               
2025-12-04T12:28:18.6792309Z     Pool 1                   
2025-12-04T12:28:18.6792444Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-12-04T12:28:18.6792606Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6792759Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6792919Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6793082Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6793246Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6793404Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6793542Z     Pool 2                   
2025-12-04T12:28:18.6793673Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-12-04T12:28:18.6793833Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6794024Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6794194Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6794366Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6794540Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6794706Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6794853Z     Pool 3                   
2025-12-04T12:28:18.6795017Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-12-04T12:28:18.6795174Z       Size:                    268419072(0xfffc000) KB            
2025-12-04T12:28:18.6795330Z       Allocatable:             TRUE                               
2025-12-04T12:28:18.6795498Z       Alloc Granule:           4KB                                
2025-12-04T12:28:18.6795669Z       Alloc Recommended Granule:2048KB                             
2025-12-04T12:28:18.6795846Z       Alloc Alignment:         4KB                                
2025-12-04T12:28:18.6796017Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6796167Z     Pool 4                   
2025-12-04T12:28:18.6796300Z       Segment:                 GROUP                              
2025-12-04T12:28:18.6796452Z       Size:                    64(0x40) KB                        
2025-12-04T12:28:18.6796608Z       Allocatable:             FALSE                              
2025-12-04T12:28:18.6796814Z       Alloc Granule:           0KB                                
2025-12-04T12:28:18.6796987Z       Alloc Recommended Granule:0KB                                
2025-12-04T12:28:18.6797154Z       Alloc Alignment:         0KB                                
2025-12-04T12:28:18.6797324Z       Accessible by all:       FALSE                              
2025-12-04T12:28:18.6797476Z   ISA Info:                
2025-12-04T12:28:18.6797590Z     ISA 1                    
2025-12-04T12:28:18.6797738Z       Name:                    amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
2025-12-04T12:28:18.6797913Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:28:18.6798084Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:28:18.6798255Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6798435Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6798602Z       Fast f16:                TRUE                               
2025-12-04T12:28:18.6798771Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6798926Z       Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6799063Z         x                        1024(0x400)                        
2025-12-04T12:28:18.6799204Z         y                        1024(0x400)                        
2025-12-04T12:28:18.6799350Z         z                        1024(0x400)                        
2025-12-04T12:28:18.6799506Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6799656Z       Grid Max Size per Dimension:
2025-12-04T12:28:18.6799788Z         x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6799932Z         y                        65535(0xffff)                      
2025-12-04T12:28:18.6800072Z         z                        65535(0xffff)                      
2025-12-04T12:28:18.6800228Z       FBarrier Max Size:       32                                 
2025-12-04T12:28:18.6800378Z     ISA 2                    
2025-12-04T12:28:18.6800528Z       Name:                    amdgcn-amd-amdhsa--gfx9-4-generic:sramecc+:xnack-
2025-12-04T12:28:18.6800709Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-12-04T12:28:18.6800882Z       Profiles:                HSA_PROFILE_BASE                   
2025-12-04T12:28:18.6801052Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6801288Z       Default Rounding Mode:   NEAR                               
2025-12-04T12:28:18.6801454Z       Fast f16:                TRUE                               
2025-12-04T12:28:18.6801616Z       Workgroup Max Size:      1024(0x400)                        
2025-12-04T12:28:18.6801768Z       Workgroup Max Size per Dimension:
2025-12-04T12:28:18.6801904Z         x                        1024(0x400)                        
2025-12-04T12:28:18.6802042Z         y                        1024(0x400)                        
2025-12-04T12:28:18.6802216Z         z                        1024(0x400)                        
2025-12-04T12:28:18.6802366Z       Grid Max Size:           4294967295(0xffffffff)             
2025-12-04T12:28:18.6802516Z       Grid Max Size per Dimension:
2025-12-04T12:28:18.6802647Z         x                        2147483647(0x7fffffff)             
2025-12-04T12:28:18.6802779Z         y                        65535(0xffff)                      
2025-12-04T12:28:18.6802918Z         z                        65535(0xffff)                      
2025-12-04T12:28:18.6803066Z       FBarrier Max Size:       32                                 
2025-12-04T12:28:18.6803205Z *** Done ***             
2025-12-04T12:28:18.6803317Z + rocminfo
2025-12-04T12:28:18.6803415Z + grep -E 'Name:.*\sgfx|Marketing'
2025-12-04T12:28:18.7660401Z   Marketing Name:          AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:28:18.7670115Z   Marketing Name:          AMD EPYC 9575F 64-Core Processor   
2025-12-04T12:28:18.7670754Z   Name:                    gfx942                             
2025-12-04T12:28:18.7671131Z   Marketing Name:          AMD Radeon Graphics                
2025-12-04T12:28:18.7671451Z   Name:                    gfx942                             
2025-12-04T12:28:18.7671758Z   Marketing Name:          AMD Radeon Graphics                
2025-12-04T12:28:18.7672072Z   Name:                    gfx942                             
2025-12-04T12:28:18.7672396Z   Marketing Name:          AMD Radeon Graphics                
2025-12-04T12:28:18.7672698Z   Name:                    gfx942                             
2025-12-04T12:28:18.7672998Z   Marketing Name:          AMD Radeon Graphics                
2025-12-04T12:28:18.7744012Z + MAYBE_ROCM=rocm/
2025-12-04T12:28:18.7744266Z + [[ linux-noble-rocm-py3.12-mi300 == *xpu* ]]
2025-12-04T12:28:18.7744552Z + [[ linux-noble-rocm-py3.12-mi300 != *-bazel-* ]]
2025-12-04T12:28:18.7744817Z + pip_install ninja==1.10.2
2025-12-04T12:28:18.7745086Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-12-04T12:28:18.7745431Z + python3 -m pip install --progress-bar off ninja==1.10.2
2025-12-04T12:28:18.9651888Z Collecting ninja==1.10.2
2025-12-04T12:28:18.9895703Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2025-12-04T12:28:18.9990599Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2025-12-04T12:28:19.0940135Z Installing collected packages: ninja
2025-12-04T12:28:19.0940587Z   Attempting uninstall: ninja
2025-12-04T12:28:19.0954409Z     Found existing installation: ninja 1.11.1.4
2025-12-04T12:28:19.0963776Z     Uninstalling ninja-1.11.1.4:
2025-12-04T12:28:19.0989185Z       Successfully uninstalled ninja-1.11.1.4
2025-12-04T12:28:19.1072562Z Successfully installed ninja-1.10.2
2025-12-04T12:28:19.1417711Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-12-04T12:28:19.1419797Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.12/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-12-04T12:28:19.1420989Z + [[ linux-noble-rocm-py3.12-mi300 == *aarch64* ]]
2025-12-04T12:28:19.1421936Z + [[ linux-noble-rocm-py3.12-mi300 == *asan* ]]
2025-12-04T12:28:19.1422333Z + [[ linux-noble-rocm-py3.12-mi300 == *-debug* ]]
2025-12-04T12:28:19.1422576Z + [[ linux-noble-rocm-py3.12-mi300 != *-bazel-* ]]
2025-12-04T12:28:19.1422980Z + echo 'We are not in debug mode: linux-noble-rocm-py3.12-mi300. Expect the assertion to pass'
2025-12-04T12:28:19.1423459Z We are not in debug mode: linux-noble-rocm-py3.12-mi300. Expect the assertion to pass
2025-12-04T12:28:19.1423805Z + cd test
2025-12-04T12:28:19.1424071Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2025-12-04T12:28:20.0304356Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2025-12-04T12:28:20.0304694Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2025-12-04T12:28:20.0305004Z + [[ distributed == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]]
2025-12-04T12:28:20.0307494Z + DYNAMO_BENCHMARK_FLAGS=()
2025-12-04T12:28:20.0307772Z + [[ distributed == *pr_time_benchmarks* ]]
2025-12-04T12:28:20.0308042Z + [[ distributed == *dynamo_eager* ]]
2025-12-04T12:28:20.0308303Z + [[ distributed == *aot_eager* ]]
2025-12-04T12:28:20.0308535Z + [[ distributed == *aot_inductor* ]]
2025-12-04T12:28:20.0308780Z + [[ distributed == *max_autotune_inductor* ]]
2025-12-04T12:28:20.0309027Z + [[ distributed == *inductor* ]]
2025-12-04T12:28:20.0309250Z + [[ distributed == *dynamic* ]]
2025-12-04T12:28:20.0309476Z + [[ distributed == *cpu* ]]
2025-12-04T12:28:20.0309685Z + [[ distributed == *xpu* ]]
2025-12-04T12:28:20.0309929Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2025-12-04T12:28:20.0323034Z + [[ linux-noble-rocm-py3.12-mi300 == *libtorch* ]]
2025-12-04T12:28:20.0324518Z + [[ linux-noble-rocm-py3.12-mi300 == *-bazel-* ]]
2025-12-04T12:28:20.0324876Z + cd test
2025-12-04T12:28:20.0325124Z + python -c 'import torch; print(torch.__config__.show())'
2025-12-04T12:28:20.8003647Z PyTorch built with:
2025-12-04T12:28:20.8003860Z   - GCC 11.5
2025-12-04T12:28:20.8003982Z   - C++ Version: 201703
2025-12-04T12:28:20.8004285Z   - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-12-04T12:28:20.8004598Z   - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-12-04T12:28:20.8004798Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-12-04T12:28:20.8004950Z   - LAPACK is enabled (usually provided by MKL)
2025-12-04T12:28:20.8005098Z   - NNPACK is enabled
2025-12-04T12:28:20.8005227Z   - CPU capability usage: AVX512
2025-12-04T12:28:20.8005362Z   - HIP Runtime 7.1.25424
2025-12-04T12:28:20.8005476Z   - MIOpen 3.5.1
2025-12-04T12:28:20.8005597Z   - Magma 2.9.0
2025-12-04T12:28:20.8007498Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_FBGEMM_GENAI -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 
2025-12-04T12:28:20.8009302Z 
2025-12-04T12:28:21.0099556Z + cd test
2025-12-04T12:28:21.0099817Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2025-12-04T12:28:21.6722846Z ATen/Parallel:
2025-12-04T12:28:21.6723173Z 	at::get_num_threads() : 128
2025-12-04T12:28:21.6724054Z 	at::get_num_interop_threads() : 128
2025-12-04T12:28:21.6724325Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-12-04T12:28:21.6724576Z 	omp_get_max_threads() : 128
2025-12-04T12:28:21.6725028Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-12-04T12:28:21.6725477Z 	mkl_get_max_threads() : 128
2025-12-04T12:28:21.6725796Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-12-04T12:28:21.6726155Z std::thread::hardware_concurrency() : 128
2025-12-04T12:28:21.6726556Z Environment variables:
2025-12-04T12:28:21.6726920Z 	OMP_NUM_THREADS : [not set]
2025-12-04T12:28:21.6727144Z 	MKL_NUM_THREADS : [not set]
2025-12-04T12:28:21.6727374Z ATen parallel backend: OpenMP
2025-12-04T12:28:21.6727527Z 
2025-12-04T12:28:21.8637167Z + [[ distributed == *numpy_2* ]]
2025-12-04T12:28:21.8637522Z + [[ linux-noble-rocm-py3.12-mi300 == *aarch64* ]]
2025-12-04T12:28:21.8637833Z + [[ distributed == *backward* ]]
2025-12-04T12:28:21.8638131Z + [[ distributed == *libtorch_agnostic_targetting* ]]
2025-12-04T12:28:21.8638427Z + [[ distributed == *xla* ]]
2025-12-04T12:28:21.8638664Z + [[ distributed == *vllm* ]]
2025-12-04T12:28:21.8638909Z + [[ distributed == *executorch* ]]
2025-12-04T12:28:21.8639175Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2025-12-04T12:28:21.8639451Z + [[ distributed == \q\u\a\n\t\i\z\a\t\i\o\n ]]
2025-12-04T12:28:21.8639753Z + [[ linux-noble-rocm-py3.12-mi300 == *libtorch* ]]
2025-12-04T12:28:21.8640046Z + [[ distributed == distributed ]]
2025-12-04T12:28:21.8640295Z + test_distributed
2025-12-04T12:28:21.8640524Z + echo 'Testing distributed python tests'
2025-12-04T12:28:21.8640797Z Testing distributed python tests
2025-12-04T12:28:21.8641140Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose
2025-12-04T12:28:23.4310354Z Excluding distributed/rpc/test_faulty_agent on ROCm
2025-12-04T12:28:23.4310935Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm
2025-12-04T12:28:23.4311654Z Excluding distributed/rpc/test_share_memory on ROCm
2025-12-04T12:28:23.4312166Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm
2025-12-04T12:28:24.2515851Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-12-04T12:28:24.6599642Z Ignoring disabled issues:  ['']
2025-12-04T12:28:24.6653257Z Found test times from artifacts
2025-12-04T12:28:24.6825606Z Found test times from artifacts
2025-12-04T12:28:24.6830938Z Running all tests
2025-12-04T12:28:24.6877547Z Running parallel tests on 1 processes
2025-12-04T12:28:24.6879338Z Name: tests to run (est. time: 116.42min)
2025-12-04T12:28:24.6879797Z   Serial tests (99):
2025-12-04T12:28:24.6880116Z     distributed/tensor/test_dtensor_export 1/1
2025-12-04T12:28:24.6880403Z     distributed/_tools/test_fake_collectives 1/1
2025-12-04T12:28:24.6880675Z     distributed/test_control_collectives 1/1
2025-12-04T12:28:24.6880920Z     distributed/test_collective_utils 1/1
2025-12-04T12:28:24.6881222Z     distributed/algorithms/quantization/test_quantization 1/1
2025-12-04T12:28:24.6881510Z     distributed/test_c10d_object_collectives 1/1
2025-12-04T12:28:24.6881807Z     distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1
2025-12-04T12:28:24.6882086Z     distributed/algorithms/test_join 1/1
2025-12-04T12:28:24.6882336Z     distributed/tensor/debug/test_op_coverage 1/1
2025-12-04T12:28:24.6882657Z     distributed/tensor/parallel/test_micro_pipeline_tp 1/1
2025-12-04T12:28:24.6882925Z     distributed/tensor/test_op_schema 1/1
2025-12-04T12:28:24.6883179Z     distributed/checkpoint/test_nested_dict 1/1
2025-12-04T12:28:24.6883469Z     distributed/checkpoint/test_consolidate_hf_safetensors 1/1
2025-12-04T12:28:24.6883750Z     distributed/_tools/test_mod_tracker 1/1
2025-12-04T12:28:24.6884007Z     distributed/_shard/sharded_tensor/test_logger 1/1
2025-12-04T12:28:24.6884268Z     distributed/tensor/test_dtensor_compile 1/1
2025-12-04T12:28:24.6884514Z     distributed/pipelining/test_pipe 1/1
2025-12-04T12:28:24.6885043Z     distributed/test_compute_comm_reordering 1/1
2025-12-04T12:28:24.6885289Z     distributed/test_p2p_ipc 1/1
2025-12-04T12:28:24.6885507Z     distributed/tensor/test_common_rules 1/1
2025-12-04T12:28:24.6885764Z     distributed/checkpoint/test_hf_safetensor_e2e 1/1
2025-12-04T12:28:24.6886013Z     distributed/tensor/test_dynamic 1/1
2025-12-04T12:28:24.6886257Z     distributed/checkpoint/e2e/test_fsdp_ep 1/1
2025-12-04T12:28:24.6886506Z     distributed/pipelining/test_unflatten 1/1
2025-12-04T12:28:24.6886974Z     distributed/tensor/test_dtensor_testbase 1/1
2025-12-04T12:28:24.6887228Z     distributed/tensor/test_redistribute 1/2
2025-12-04T12:28:24.6887472Z     distributed/_tools/test_sac_estimator 1/1
2025-12-04T12:28:24.6887706Z     distributed/_tools/test_memory_tracker 1/1
2025-12-04T12:28:24.6887982Z     distributed/checkpoint/_experimental/test_builder 1/1
2025-12-04T12:28:24.6888286Z     distributed/_composable/test_replicate_with_fsdp 1/1
2025-12-04T12:28:24.6888537Z     distributed/test_backends 1/1
2025-12-04T12:28:24.6888765Z     distributed/tensor/test_experimental_ops 1/1
2025-12-04T12:28:24.6889035Z     distributed/checkpoint/test_async_process_executor 1/1
2025-12-04T12:28:24.6889285Z     distributed/test_nvshmem 1/1
2025-12-04T12:28:24.6889492Z     distributed/tensor/test_attention 1/1
2025-12-04T12:28:24.6889750Z     distributed/tensor/test_convolution_ops 1/1
2025-12-04T12:28:24.6889993Z     distributed/checkpoint/test_save_load_api 1/1
2025-12-04T12:28:24.6890218Z     distributed/test_debug 1/1
2025-12-04T12:28:24.6890441Z     distributed/test_overlap_bucketing_unit 1/1
2025-12-04T12:28:24.6890724Z     distributed/checkpoint/_experimental/test_checkpoint_writer 1/1
2025-12-04T12:28:24.6891018Z     distributed/optim/test_named_optimizer 1/1
2025-12-04T12:28:24.6891288Z     distributed/checkpoint/_experimental/test_checkpointer 1/1
2025-12-04T12:28:24.6891560Z     distributed/tensor/test_api 1/1
2025-12-04T12:28:24.6891781Z     distributed/checkpoint/test_traverse 1/1
2025-12-04T12:28:24.6892009Z     distributed/tensor/test_random_ops 1/1
2025-12-04T12:28:24.6892264Z     distributed/checkpoint/_experimental/test_staging 1/1
2025-12-04T12:28:24.6892565Z     distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1
2025-12-04T12:28:24.6892805Z     distributed/launcher/test_api 1/1
2025-12-04T12:28:24.6893001Z     distributed/elastic/multiprocessing/test_api 1/1
2025-12-04T12:28:24.6893186Z     distributed/fsdp/test_shard_utils 1/1
2025-12-04T12:28:24.6893372Z     distributed/tensor/experimental/test_local_map 1/1
2025-12-04T12:28:24.6893562Z     distributed/test_local_tensor 1/1
2025-12-04T12:28:24.6893752Z     distributed/_composable/fsdp/test_fully_shard_state 1/1
2025-12-04T12:28:24.6893960Z     distributed/checkpoint/test_tp_checkpoint 1/1
2025-12-04T12:28:24.6894143Z     distributed/tensor/test_optimizers 1/1
2025-12-04T12:28:24.6894333Z     distributed/checkpoint/test_dtensor_resharding 1/1
2025-12-04T12:28:24.6894518Z     distributed/test_symmetric_memory 1/1
2025-12-04T12:28:24.6894719Z     distributed/_composable/test_replicate_with_compiler 1/1
2025-12-04T12:28:24.6894949Z     distributed/_composable/fsdp/test_fully_shard_autograd 1/1
2025-12-04T12:28:24.6895205Z     distributed/_composable/test_composability/test_2d_composability 1/1
2025-12-04T12:28:24.6895436Z     distributed/fsdp/test_fsdp_optim_state 1/1
2025-12-04T12:28:24.6895614Z     distributed/fsdp/test_fsdp_state_dict 2/2
2025-12-04T12:28:24.6895787Z     distributed/test_distributed_spawn 2/7
2025-12-04T12:28:24.6895963Z     distributed/test_distributed_spawn 5/7
2025-12-04T12:28:24.6896131Z     distributed/fsdp/test_fsdp_input 1/1
2025-12-04T12:28:24.6896320Z     distributed/_composable/fsdp/test_fully_shard_comm 1/1
2025-12-04T12:28:24.6896535Z     distributed/checkpoint/test_file_system_checkpoint 1/1
2025-12-04T12:28:24.6896729Z     distributed/test_composability 1/1
2025-12-04T12:28:24.6896933Z     distributed/test_multi_threaded_pg 1/1
2025-12-04T12:28:24.6897190Z     distributed/_composable/fsdp/test_fully_shard_extensions 1/1
2025-12-04T12:28:24.6897435Z     distributed/checkpoint/test_file_system_checkpoint_cpu 1/1
2025-12-04T12:28:24.6897637Z     distributed/fsdp/test_wrap 1/1
2025-12-04T12:28:24.6897818Z     distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1
2025-12-04T12:28:24.6898015Z     distributed/fsdp/test_fsdp_tp_integration 1/1
2025-12-04T12:28:24.6898224Z     distributed/_shard/sharded_optim/test_sharded_optim 1/1
2025-12-04T12:28:24.6898454Z     distributed/_composable/fsdp/test_fully_shard_state_dict 1/1
2025-12-04T12:28:24.6898721Z     distributed/fsdp/test_fsdp_clip_grad_norm 1/1
2025-12-04T12:28:24.6898914Z     distributed/fsdp/test_fsdp_multiple_forward 1/1
2025-12-04T12:28:24.6899105Z     distributed/nn/jit/test_instantiator 1/1
2025-12-04T12:28:24.6899304Z     distributed/_shard/sharding_spec/test_sharding_spec 1/1
2025-12-04T12:28:24.6899499Z     distributed/fsdp/test_fsdp_misc 1/1
2025-12-04T12:28:24.6899696Z     distributed/_composable/fsdp/test_fully_shard_memory 1/1
2025-12-04T12:28:24.6899922Z     distributed/optim/test_zero_redundancy_optimizer 1/1
2025-12-04T12:28:24.6900112Z     distributed/fsdp/test_fsdp_core 2/3
2025-12-04T12:28:24.6900298Z     distributed/fsdp/test_distributed_checkpoint 1/1
2025-12-04T12:28:24.6900485Z     distributed/test_c10d_spawn_nccl 1/1
2025-12-04T12:28:24.6900650Z     distributed/test_c10d_gloo 2/2
2025-12-04T12:28:24.6900807Z     distributed/test_launcher 1/1
2025-12-04T12:28:24.6900962Z     distributed/test_store 1/1
2025-12-04T12:28:24.6901110Z     distributed/test_c10d_nccl 1/3
2025-12-04T12:28:24.6901272Z     distributed/test_c10d_ops_nccl 1/1
2025-12-04T12:28:24.6901441Z     distributed/elastic/events/lib_test 1/1
2025-12-04T12:28:24.6901617Z     distributed/elastic/metrics/api_test 1/1
2025-12-04T12:28:24.6901808Z     distributed/elastic/multiprocessing/api_test 1/1
2025-12-04T12:28:24.6902019Z     distributed/elastic/timer/local_timer_example 1/1
2025-12-04T12:28:24.6902219Z     distributed/elastic/timer/local_timer_test 1/1
2025-12-04T12:28:24.6902418Z     distributed/elastic/utils/distributed_test 1/1
2025-12-04T12:28:24.6902607Z     distributed/elastic/utils/logging_test 1/1
2025-12-04T12:28:24.6902783Z     distributed/elastic/utils/util_test 1/1
2025-12-04T12:28:24.6902943Z   Parallel tests (0):
2025-12-04T12:28:24.6903084Z Name: excluded (est. time: 0.0min)
2025-12-04T12:28:24.6903234Z   Serial tests (0):
2025-12-04T12:28:24.6903358Z   Parallel tests (0):
2025-12-04T12:28:24.6903559Z Running distributed/tensor/test_dtensor_export 1/1 ... [2025-12-04 12:28:24.688210][2235129.145530995]
2025-12-04T12:28:24.6903783Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:28:24.6904227Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_dtensor_export.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:28:24.688440]
2025-12-04T12:28:30.7129114Z 
2025-12-04T12:28:30.7130006Z distributed/tensor/test_dtensor_export 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_export_1.1_dbdb5be8f0bd41d5_.log
2025-12-04T12:28:30.7144970Z Running 9 items in this shard: test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_annotate_aot_export_joint_with_descriptors_alone, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_dtensor_data_dependent_index_and_slice, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_dynamic_shapes_export_fn_with_answer0, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_einsum_dtensor_export_export_fn0, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_export_parallelize_module_with_dtensor_input_export_fn0, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_export_parallelize_module_with_dtensor_input_export_fn1, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_flex_attention_dtensor_export_export_fn0, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_strict_export_parallelize_module_with_dtensor_input, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_union_typed_annotation
2025-12-04T12:28:30.7148214Z 
2025-12-04T12:28:30.7148497Z Finished distributed/tensor/test_dtensor_export 1/1 ... [2025-12-04 12:28:30.712785][2235135.170103124], took 0.10min
2025-12-04T12:28:30.7149293Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:28:32.7576365Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:28:32.7577165Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-12-04T12:28:32.7577581Z Uploading artifacts took 0.00 seconds
2025-12-04T12:28:32.7578131Z Running distributed/_tools/test_fake_collectives 1/1 ... [2025-12-04 12:28:32.757366][2235137.214684379]
2025-12-04T12:28:32.7578612Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:28:32.7580197Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_tools/test_fake_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:28:32.757648]
2025-12-04T12:28:35.1258087Z 
2025-12-04T12:28:35.1259322Z distributed/_tools/test_fake_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_fake_collectives_1.1_67d08f7458b736f9_.log
2025-12-04T12:28:35.1260749Z Running 1 items in this shard: test/distributed/_tools/test_fake_collectives.py::TestFakeCollectives::test_collectives
2025-12-04T12:28:35.1261268Z 
2025-12-04T12:28:35.1261648Z Finished distributed/_tools/test_fake_collectives 1/1 ... [2025-12-04 12:28:35.125393][2235139.58271349], took 0.04min
2025-12-04T12:28:35.1262966Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:28:35.1269566Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:28:35.1272217Z Running distributed/test_control_collectives 1/1 ... [2025-12-04 12:28:35.127123][2235139.584446712]
2025-12-04T12:28:35.1272554Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:28:35.1276343Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_control_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:28:35.127320]
2025-12-04T12:28:37.2452195Z 
2025-12-04T12:28:37.2453133Z distributed/test_control_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_control_collectives_1.1_d99ce806775c3f76_.log
2025-12-04T12:28:37.2456686Z Running 13 items in this shard: test/distributed/test_control_collectives.py::TestCollectives::test_all_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_barrier, test/distributed/test_control_collectives.py::TestCollectives::test_barrier_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_gather, test/distributed/test_control_collectives.py::TestCollectives::test_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_scatter, test/distributed/test_control_collectives.py::TestCollectives::test_scatter_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_simple_user_func, test/distributed/test_control_collectives.py::TestCollectives::test_unique
2025-12-04T12:28:37.2459420Z 
2025-12-04T12:28:37.2459626Z Finished distributed/test_control_collectives 1/1 ... [2025-12-04 12:28:37.244841][2235141.702160085], took 0.04min
2025-12-04T12:28:37.2460307Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:28:37.2465859Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:28:37.2472031Z Running distributed/test_collective_utils 1/1 ... [2025-12-04 12:28:37.246860][2235141.704183002]
2025-12-04T12:28:37.2472296Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:28:37.2472812Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:28:37.247084]
2025-12-04T12:28:56.6914190Z 
2025-12-04T12:28:56.6915395Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_e1433662e95f8204_.log
2025-12-04T12:28:56.6919292Z Running 9 items in this shard: test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result_no_pg, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result_raises_exceptions_from_func, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result_no_pg, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result_raises_exceptions_from_func, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_check_rng_sync_device_cpu, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_check_rng_sync_device_cuda, test/distributed/test_collective_utils.py::TestUtils::test_summarize_ranks
2025-12-04T12:28:56.6923125Z 
2025-12-04T12:28:56.6923617Z Finished distributed/test_collective_utils 1/1 ... [2025-12-04 12:28:56.691104][2235161.148425538], took 0.32min
2025-12-04T12:28:56.6924724Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:28:56.6926476Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:28:56.6928014Z Running distributed/algorithms/quantization/test_quantization 1/1 ... [2025-12-04 12:28:56.692683][2235161.150006873]
2025-12-04T12:28:56.6929738Z MPI not available -- MPI backend tests will be skipped
2025-12-04T12:28:56.6930940Z Running distributed tests for the test backend with env init_method
2025-12-04T12:28:56.6931396Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:28:56.6933382Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:28:56.693188]
2025-12-04T12:28:58.5212305Z 
2025-12-04T12:28:58.5213746Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_57d203a22015377d_.log
2025-12-04T12:28:58.5214869Z Running 0 items in this shard:
2025-12-04T12:28:58.5215085Z 
2025-12-04T12:28:58.5218171Z Running distributed tests for the test backend with file init_method
2025-12-04T12:28:58.5220819Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:28:58.5222438Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:28:58.521980]
2025-12-04T12:29:00.3478460Z 
2025-12-04T12:29:00.3479471Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_8cac3a0592bf6c1a_.log
2025-12-04T12:29:00.3480883Z Running 0 items in this shard:
2025-12-04T12:29:00.3481090Z 
2025-12-04T12:29:00.3487246Z Running distributed tests for the nccl backend with env init_method
2025-12-04T12:29:00.3487716Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:29:00.3490128Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:29:00.348756]
2025-12-04T12:29:24.3168845Z 
2025-12-04T12:29:24.3169809Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_27cff313179f3e84_.log
2025-12-04T12:29:24.3171299Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-12-04T12:29:24.3172644Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16
2025-12-04T12:29:24.3173057Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16
2025-12-04T12:29:24.3173469Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16
2025-12-04T12:29:24.3173871Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16
2025-12-04T12:29:24.3174291Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16
2025-12-04T12:29:24.3174717Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-12-04T12:29:24.3174950Z 
2025-12-04T12:29:24.3175059Z Running distributed tests for the nccl backend with file init_method
2025-12-04T12:29:24.3175238Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:29:24.3175712Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:29:24.317365]
2025-12-04T12:29:46.6435073Z 
2025-12-04T12:29:46.6438307Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_a412695cded491f4_.log
2025-12-04T12:29:46.6440347Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-12-04T12:29:46.6441785Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16
2025-12-04T12:29:46.6442187Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16
2025-12-04T12:29:46.6444136Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16
2025-12-04T12:29:46.6444540Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16
2025-12-04T12:29:46.6444956Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16
2025-12-04T12:29:46.6445380Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-12-04T12:29:46.6445614Z 
2025-12-04T12:29:46.6445713Z Running distributed tests for the gloo backend with env init_method
2025-12-04T12:29:46.6445890Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:29:46.6446364Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:29:46.644295]
2025-12-04T12:30:04.4706184Z 
2025-12-04T12:30:04.4707124Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_b0b2201a4ba70adc_.log
2025-12-04T12:30:04.4708620Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-12-04T12:30:04.4709870Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16
2025-12-04T12:30:04.4710314Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16
2025-12-04T12:30:04.4710728Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16
2025-12-04T12:30:04.4711132Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16
2025-12-04T12:30:04.4712084Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16
2025-12-04T12:30:04.4712511Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-12-04T12:30:04.4712742Z 
2025-12-04T12:30:04.4712838Z Running distributed tests for the gloo backend with file init_method
2025-12-04T12:30:04.4713323Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:30:04.4716099Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:30:04.471428]
2025-12-04T12:30:21.6900079Z 
2025-12-04T12:30:21.6901092Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_16622fa756ab4077_.log
2025-12-04T12:30:21.6902530Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-12-04T12:30:21.6903818Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16
2025-12-04T12:30:21.6904250Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16
2025-12-04T12:30:21.6904649Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16
2025-12-04T12:30:21.6905044Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16
2025-12-04T12:30:21.6905453Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16
2025-12-04T12:30:21.6905872Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-12-04T12:30:21.6906100Z 
2025-12-04T12:30:21.6909156Z Finished distributed/algorithms/quantization/test_quantization 1/1 ... [2025-12-04 12:30:21.690039][2235246.147359027], took 1.42min
2025-12-04T12:30:21.6909643Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:30:21.6914948Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:30:21.6917864Z Running distributed/test_c10d_object_collectives 1/1 ... [2025-12-04 12:30:21.691661][2235246.148985381]
2025-12-04T12:30:21.6919190Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:30:21.6920216Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:30:21.691839]
2025-12-04T12:31:02.8183907Z 
2025-12-04T12:31:02.8185130Z distributed/test_c10d_object_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_object_collectives_1.1_e7fba64e0baa3cc3_.log
2025-12-04T12:31:02.8189876Z Running 9 items in this shard: test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_broadcast_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_scatter_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_send_recv_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_broadcast_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_scatter_object
2025-12-04T12:31:02.8193009Z 
2025-12-04T12:31:02.8193293Z Finished distributed/test_c10d_object_collectives 1/1 ... [2025-12-04 12:31:02.818026][2235287.275343401], took 0.69min
2025-12-04T12:31:02.8194204Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:31:02.8194992Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:31:02.8198576Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 ... [2025-12-04 12:31:02.819734][2235287.277058213]
2025-12-04T12:31:02.8199030Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:31:02.8203243Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:31:02.819923]
2025-12-04T12:31:33.0805985Z 
2025-12-04T12:31:33.0807355Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_2987461eff93faaf_.log
2025-12-04T12:31:33.0810684Z Running 6 items in this shard: test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_allreduce_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_fp16compress_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_noop_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_channel_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_tensor_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_is_last_hook
2025-12-04T12:31:33.0813112Z 
2025-12-04T12:31:33.0813432Z Finished distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 ... [2025-12-04 12:31:33.080248][2235317.537567175], took 0.50min
2025-12-04T12:31:33.0814349Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:31:33.0821356Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:31:33.0824809Z Running distributed/algorithms/test_join 1/1 ... [2025-12-04 12:31:33.082381][2235317.539705891]
2025-12-04T12:31:33.0825115Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:31:33.0827772Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/algorithms/test_join.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:31:33.082593]
2025-12-04T12:32:14.0626267Z 
2025-12-04T12:32:14.0627121Z distributed/algorithms/test_join 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.test_join_1.1_703b148d57da3f37_.log
2025-12-04T12:32:14.0629934Z Running 9 items in this shard: test/distributed/algorithms/test_join.py::TestJoin::test_join_kwargs, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables_throw, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_main_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_post_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_throw
2025-12-04T12:32:14.0633639Z 
2025-12-04T12:32:14.0634017Z Finished distributed/algorithms/test_join 1/1 ... [2025-12-04 12:32:14.062404][2235358.519722786], took 0.68min
2025-12-04T12:32:14.0635270Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:32:14.0641799Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:32:14.0645033Z Running distributed/tensor/debug/test_op_coverage 1/1 ... [2025-12-04 12:32:14.064388][2235358.521712644]
2025-12-04T12:32:14.0645517Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:32:14.0647689Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:32:14.064601]
2025-12-04T12:32:16.5346723Z 
2025-12-04T12:32:16.5348059Z distributed/tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.debug.test_op_coverage_1.1_01406ac49c7e48fe_.log
2025-12-04T12:32:16.5349458Z Running 1 items in this shard: test/distributed/tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp
2025-12-04T12:32:16.5350017Z 
2025-12-04T12:32:16.5350411Z Finished distributed/tensor/debug/test_op_coverage 1/1 ... [2025-12-04 12:32:16.534425][2235360.991746966], took 0.04min
2025-12-04T12:32:16.5352669Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:32:16.5359645Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:32:16.5362151Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2025-12-04 12:32:16.536128][2235360.993453209]
2025-12-04T12:32:16.5362465Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:32:16.5364358Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:32:16.536306]
2025-12-04T12:32:37.7323552Z 
2025-12-04T12:32:37.7324894Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_23627a1a354e24da_.log
2025-12-04T12:32:37.7342638Z Running 44 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTP4GPUTest::test_extra_collectives
2025-12-04T12:32:37.7353096Z 
2025-12-04T12:32:37.7353282Z Finished distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2025-12-04 12:32:37.731949][2235382.189269672], took 0.35min
2025-12-04T12:32:37.7353817Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:32:37.7354259Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:32:37.7354501Z Running distributed/tensor/test_op_schema 1/1 ... [2025-12-04 12:32:37.734021][2235382.191344968]
2025-12-04T12:32:37.7354703Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:32:37.7355111Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_op_schema.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:32:37.734222]
2025-12-04T12:32:39.8517440Z 
2025-12-04T12:32:39.8518622Z distributed/tensor/test_op_schema 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_op_schema_1.1_49197acd2c626b34_.log
2025-12-04T12:32:39.8520364Z Running 2 items in this shard: test/distributed/tensor/test_op_schema.py::TestOpSchema::test_equality_checks_lists_of_dtensor_spec, test/distributed/tensor/test_op_schema.py::TestOpSchema::test_equality_respects_static_attributes
2025-12-04T12:32:39.8521428Z 
2025-12-04T12:32:39.8521800Z Finished distributed/tensor/test_op_schema 1/1 ... [2025-12-04 12:32:39.851407][2235384.308726345], took 0.04min
2025-12-04T12:32:39.8525199Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:32:39.8533018Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:32:39.8535908Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2025-12-04 12:32:39.853448][2235384.310772222]
2025-12-04T12:32:39.8536249Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:32:39.8537941Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:32:39.853628]
2025-12-04T12:32:42.0216177Z 
2025-12-04T12:32:42.0217380Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_9f88d7eb0b1d917f_.log
2025-12-04T12:32:42.0219204Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping
2025-12-04T12:32:42.0219865Z 
2025-12-04T12:32:42.0220181Z Finished distributed/checkpoint/test_nested_dict 1/1 ... [2025-12-04 12:32:42.021223][2235386.47854403], took 0.04min
2025-12-04T12:32:42.0222729Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:32:42.0230320Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:32:42.0234556Z Running distributed/checkpoint/test_consolidate_hf_safetensors 1/1 ... [2025-12-04 12:32:42.023186][2235386.480510439]
2025-12-04T12:32:42.0234966Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:32:42.0235769Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_consolidate_hf_safetensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:32:42.023384]
2025-12-04T12:33:09.5826110Z 
2025-12-04T12:33:09.5827752Z distributed/checkpoint/test_consolidate_hf_safetensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_consolidate_hf_safetensors_1.1_4db569985bb1e23b_.log
2025-12-04T12:33:09.5832693Z Running 7 items in this shard: test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_calculate_max_contiguous_elements_valid_cases, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_calculate_max_contiguous_elements_validations, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_consolidate_one_file_with_two_ranks, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_consolidate_to_one_file, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_consolidate_to_two_files, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_consolidate_with_two_ranks, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_write_sub_tensor_to_file_optimized
2025-12-04T12:33:09.5836086Z 
2025-12-04T12:33:09.5836463Z Finished distributed/checkpoint/test_consolidate_hf_safetensors 1/1 ... [2025-12-04 12:33:09.582283][2235414.039604201], took 0.46min
2025-12-04T12:33:09.5837633Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:33:09.5843640Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:33:09.5845040Z Running distributed/_tools/test_mod_tracker 1/1 ... [2025-12-04 12:33:09.584347][2235414.041671327]
2025-12-04T12:33:09.5845368Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:33:09.5847527Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:33:09.584542]
2025-12-04T12:33:11.6521042Z 
2025-12-04T12:33:11.6521704Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_d124228084baadd3_.log
2025-12-04T12:33:11.6522631Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks
2025-12-04T12:33:11.6523655Z 
2025-12-04T12:33:11.6523798Z Finished distributed/_tools/test_mod_tracker 1/1 ... [2025-12-04 12:33:11.651797][2235416.109119153], took 0.03min
2025-12-04T12:33:11.6528930Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:33:11.6532510Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:33:11.6534721Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2025-12-04 12:33:11.653384][2235416.110707978]
2025-12-04T12:33:11.6535120Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:33:11.6537042Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:33:11.653544]
2025-12-04T12:33:13.7718726Z 
2025-12-04T12:33:13.7720048Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_5363516baa5c8354_.log
2025-12-04T12:33:13.7721532Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger
2025-12-04T12:33:13.7722120Z 
2025-12-04T12:33:13.7722522Z Finished distributed/_shard/sharded_tensor/test_logger 1/1 ... [2025-12-04 12:33:13.771531][2235418.22885128], took 0.04min
2025-12-04T12:33:13.7726065Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:33:13.7734216Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:33:13.7736305Z Running distributed/tensor/test_dtensor_compile 1/1 ... [2025-12-04 12:33:13.773546][2235418.230870157]
2025-12-04T12:33:13.7736632Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:33:13.7738749Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:33:13.773737]
2025-12-04T12:36:34.7478196Z 
2025-12-04T12:36:34.7479376Z distributed/tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_compile_1.1_44bcc97a2337a91c_.log
2025-12-04T12:36:34.7498991Z Running 49 items in this shard: test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_basic, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_basic_export, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_cat, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_loss_parallel_log_softmax, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_recompiles, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_slice, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_requires_grad_recompile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_async, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_from_local_grad_placements_sequence_intermediate, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_from_local_grad_placements_sequence_intermediate_as_args, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_grad_placements_sequence, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_grad_placements_sequence_intermediate, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_get_local_rank_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering_graph_partition, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_embedding_redistribute, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_ca_True
2025-12-04T12:36:34.7509512Z 
2025-12-04T12:36:34.7509717Z Finished distributed/tensor/test_dtensor_compile 1/1 ... [2025-12-04 12:36:34.747495][2235619.204815212], took 3.35min
2025-12-04T12:36:34.7510321Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:36:34.7510775Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:36:34.7511032Z Running distributed/pipelining/test_pipe 1/1 ... [2025-12-04 12:36:34.749521][2235619.206845469]
2025-12-04T12:36:34.7511251Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:36:34.7511674Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:36:34.749719]
2025-12-04T12:36:37.6816148Z 
2025-12-04T12:36:37.6817559Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_c197d2d902d9b235_.log
2025-12-04T12:36:37.6819445Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2
2025-12-04T12:36:37.6820113Z 
2025-12-04T12:36:37.6820329Z Finished distributed/pipelining/test_pipe 1/1 ... [2025-12-04 12:36:37.681252][2235622.138573246], took 0.05min
2025-12-04T12:36:37.6822229Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:36:37.6831112Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:36:37.6832321Z Running distributed/test_compute_comm_reordering 1/1 ... [2025-12-04 12:36:37.683072][2235622.140397096]
2025-12-04T12:36:37.6832616Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:36:37.6834037Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:36:37.683246]
2025-12-04T12:38:17.6996447Z 
2025-12-04T12:38:17.6997551Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_b99015b14f13b4e3_.log
2025-12-04T12:38:17.7002313Z Running 9 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node_combo_kernels_False, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node_combo_kernels_True, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_inductor_default_comms_ordering, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms
2025-12-04T12:38:17.7005773Z 
2025-12-04T12:38:17.7006033Z Finished distributed/test_compute_comm_reordering 1/1 ... [2025-12-04 12:38:17.699297][2235722.156616668], took 1.67min
2025-12-04T12:38:17.7006893Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:38:17.7011576Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:38:17.7013858Z Running distributed/test_p2p_ipc 1/1 ... [2025-12-04 12:38:17.701293][2235722.158617796]
2025-12-04T12:38:17.7014188Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:38:17.7015969Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_p2p_ipc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:38:17.701474]
2025-12-04T12:38:22.4737448Z 
2025-12-04T12:38:22.4738546Z distributed/test_p2p_ipc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_p2p_ipc_1.1_89ec76b0cb7ef164_.log
2025-12-04T12:38:22.4739766Z Running 1 items in this shard: test/distributed/test_p2p_ipc.py::P2PIpcTest::test_p2p_ipc
2025-12-04T12:38:22.4740189Z 
2025-12-04T12:38:22.4740515Z Finished distributed/test_p2p_ipc 1/1 ... [2025-12-04 12:38:22.473329][2235726.930650753], took 0.08min
2025-12-04T12:38:22.4744066Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:38:22.4750521Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:38:22.4752916Z Running distributed/tensor/test_common_rules 1/1 ... [2025-12-04 12:38:22.475177][2235726.932501753]
2025-12-04T12:38:22.4753347Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:38:22.4755084Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:38:22.475363]
2025-12-04T12:38:26.4962040Z 
2025-12-04T12:38:26.4963215Z distributed/tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_common_rules_1.1_cc38bc1284ef55fe_.log
2025-12-04T12:38:26.4968594Z Running 10 items in this shard: test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion
2025-12-04T12:38:26.4972484Z 
2025-12-04T12:38:26.4972806Z Finished distributed/tensor/test_common_rules 1/1 ... [2025-12-04 12:38:26.495708][2235730.95302943], took 0.07min
2025-12-04T12:38:26.4973580Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:38:26.4974393Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:38:26.4975705Z Running distributed/checkpoint/test_hf_safetensor_e2e 1/1 ... [2025-12-04 12:38:26.497476][2235730.954801042]
2025-12-04T12:38:26.4976066Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:38:26.4977821Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_hf_safetensor_e2e.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:38:26.497646]
2025-12-04T12:38:56.6593282Z 
2025-12-04T12:38:56.6594019Z distributed/checkpoint/test_hf_safetensor_e2e 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hf_safetensor_e2e_1.1_582013219c315b54_.log
2025-12-04T12:38:56.6598736Z Running 11 items in this shard: test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestSingleRankSaveLoad::test_load, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestSingleRankSaveLoad::test_load_into_empty_dict, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestSingleRankSaveLoad::test_load_with_multiple_threads, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestSingleRankSaveLoad::test_quantized_checkpoint_loading, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestSingleRankSaveLoad::test_save, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDistributedHFSafetensorsConsolidation::test_consolidate_to_one_file, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard
2025-12-04T12:38:56.6608568Z 
2025-12-04T12:38:56.6608745Z Finished distributed/checkpoint/test_hf_safetensor_e2e 1/1 ... [2025-12-04 12:38:56.659047][2235761.116366169], took 0.50min
2025-12-04T12:38:56.6609254Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:38:56.6610839Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:38:56.6613625Z Running distributed/tensor/test_dynamic 1/1 ... [2025-12-04 12:38:56.661285][2235761.118609193]
2025-12-04T12:38:56.6613839Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:38:56.6615673Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_dynamic.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:38:56.661473]
2025-12-04T12:39:30.6282288Z 
2025-12-04T12:39:30.6284617Z distributed/tensor/test_dynamic 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dynamic_1.1_f73b6397a621e5e5_.log
2025-12-04T12:39:30.6287552Z Running 4 items in this shard: test/distributed/tensor/test_dynamic.py::TestDynamic::test_embedding_fake_tensor_cache_enabled_False, test/distributed/tensor/test_dynamic.py::TestDynamic::test_embedding_fake_tensor_cache_enabled_True, test/distributed/tensor/test_dynamic.py::TestDynamicWithLocalTensor::test_embedding_fake_tensor_cache_enabled_False, test/distributed/tensor/test_dynamic.py::TestDynamicWithLocalTensor::test_embedding_fake_tensor_cache_enabled_True
2025-12-04T12:39:30.6289498Z 
2025-12-04T12:39:30.6289770Z Finished distributed/tensor/test_dynamic 1/1 ... [2025-12-04 12:39:30.627860][2235795.085178868], took 0.57min
2025-12-04T12:39:30.6290711Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:39:30.6294856Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:39:30.6297079Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2025-12-04 12:39:30.629597][2235795.08692229]
2025-12-04T12:39:30.6297506Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:39:30.6299227Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:39:30.629777]
2025-12-04T12:39:35.3018897Z 
2025-12-04T12:39:35.3020079Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_986a47429ec203c4_.log
2025-12-04T12:39:35.3021388Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e
2025-12-04T12:39:35.3021867Z 
2025-12-04T12:39:35.3022316Z Finished distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2025-12-04 12:39:35.301508][2235799.758827289], took 0.08min
2025-12-04T12:39:35.3028514Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:39:35.3035305Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:39:35.3037586Z Running distributed/pipelining/test_unflatten 1/1 ... [2025-12-04 12:39:35.303648][2235799.760972575]
2025-12-04T12:39:35.3038068Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:39:35.3040043Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/pipelining/test_unflatten.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:39:35.303842]
2025-12-04T12:39:42.6800923Z 
2025-12-04T12:39:42.6801902Z distributed/pipelining/test_unflatten 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_unflatten_1.1_d61869ca51181915_.log
2025-12-04T12:39:42.6802449Z Running 1 items in this shard: test/distributed/pipelining/test_unflatten.py::UnflattenTestsCUDA::test_unflatten_cuda
2025-12-04T12:39:42.6802641Z 
2025-12-04T12:39:42.6802791Z Finished distributed/pipelining/test_unflatten 1/1 ... [2025-12-04 12:39:42.679662][2235807.136983113], took 0.12min
2025-12-04T12:39:42.6808139Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:39:42.6811804Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:39:42.6815827Z Running distributed/tensor/test_dtensor_testbase 1/1 ... [2025-12-04 12:39:42.681342][2235807.138666506]
2025-12-04T12:39:42.6816609Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:39:48.5561010Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_dtensor_testbase.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:39:42.681519]
2025-12-04T12:39:48.5561874Z 
2025-12-04T12:39:48.5562429Z distributed/tensor/test_dtensor_testbase 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_testbase_1.1_a3eb1abe09dc5efa_.log
2025-12-04T12:39:48.5563990Z Running 1 items in this shard: test/distributed/tensor/test_dtensor_testbase.py::DTensorTestBaseUtilCPUTest::test_dtensor_testbase_destroy_pg
2025-12-04T12:39:48.5564452Z 
2025-12-04T12:39:48.5564765Z Finished distributed/tensor/test_dtensor_testbase 1/1 ... [2025-12-04 12:39:48.555744][2235813.013062754], took 0.10min
2025-12-04T12:39:48.5571506Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:39:48.5577408Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:39:48.5580056Z Running distributed/tensor/test_redistribute 1/2 ... [2025-12-04 12:39:48.557909][2235813.015234159]
2025-12-04T12:39:48.5580390Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:39:48.5582022Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_redistribute.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:39:48.558083]
2025-12-04T12:40:56.2311387Z 
2025-12-04T12:40:56.2312198Z distributed/tensor/test_redistribute 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_redistribute_1.2_0c88288acf2c4085_.log
2025-12-04T12:40:56.2319012Z Running 25 items in this shard: test/distributed/tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard_float32, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_negative_shard_dim, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_shard_dim_change_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_shard_dim_change_float32, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_to_partial, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_uneven_sharding, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_replicate_to_partial, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward_datatype_conversion, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward_float32, test/distributed/tensor/test_redistribute.py::MultiDimRedistributeTest::test_multi_dim_mesh, test/distributed/tensor/test_redistribute.py::DistributeWithDeviceOrderTest::test_ordered_redistribute, test/distributed/tensor/test_redistribute.py::DistributeWithDeviceOrderTest::test_ordered_redistribute_for_special_placement, test/distributed/tensor/test_redistribute.py::RedistributeTestWithLocalTensor::test_partial_to_shard_float32, test/distributed/tensor/test_redistribute.py::RedistributeTestWithLocalTensor::test_redistribute_shard_dim_change_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTestWithLocalTensor::test_redistribute_shard_dim_change_float32, test/distributed/tensor/test_redistribute.py::RedistributeTestWithLocalTensor::test_replicate_to_local_partial_grad_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTestWithLocalTensor::test_replicate_to_local_partial_grad_float32, test/distributed/tensor/test_redistribute.py::RedistributeTestWithLocalTensor::test_replicate_to_shard_forward_backward, test/distributed/tensor/test_redistribute.py::RedistributeTestWithLocalTensor::test_shard_dim_alltoall_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTestWithLocalTensor::test_shard_to_replicate_forward_backward_complex64, test/distributed/tensor/test_redistribute.py::MultiDimRedistributeTestWithLocalTensor::test_redistribute_shard_dim_multi_dim_mesh, test/distributed/tensor/test_redistribute.py::DistributeWithDeviceOrderTestWithLocalTensor::test_ordered_distribute_all_combination, test/distributed/tensor/test_redistribute.py::DistributeWithDeviceOrderTestWithLocalTensor::test_shard_order_same_data_as_strided_shard
2025-12-04T12:40:56.2324697Z 
2025-12-04T12:40:56.2324845Z Finished distributed/tensor/test_redistribute 1/2 ... [2025-12-04 12:40:56.230750][2235880.688072002], took 1.13min
2025-12-04T12:40:56.2325328Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:40:56.2325760Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:40:56.2326024Z Running distributed/_tools/test_sac_estimator 1/1 ... [2025-12-04 12:40:56.232403][2235880.689727696]
2025-12-04T12:40:56.2326252Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:40:56.2326720Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_tools/test_sac_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:40:56.232576]
2025-12-04T12:41:00.4029434Z 
2025-12-04T12:41:00.4030590Z distributed/_tools/test_sac_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_estimator_1.1_106a20f5a58b0752_.log
2025-12-04T12:41:00.4032276Z Running 2 items in this shard: test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_simple_model_sac_estimation, test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_transformer_sac_estimation
2025-12-04T12:41:00.4033225Z 
2025-12-04T12:41:00.4033545Z Finished distributed/_tools/test_sac_estimator 1/1 ... [2025-12-04 12:41:00.402553][2235884.859873282], took 0.07min
2025-12-04T12:41:00.4039653Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:41:00.4046573Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:41:00.4049447Z Running distributed/_tools/test_memory_tracker 1/1 ... [2025-12-04 12:41:00.404856][2235884.862180285]
2025-12-04T12:41:00.4049784Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:41:00.4051649Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:41:00.405048]
2025-12-04T12:41:06.3288280Z 
2025-12-04T12:41:06.3289547Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_799f7bbed7567005_.log
2025-12-04T12:41:06.3290888Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model
2025-12-04T12:41:06.3291393Z 
2025-12-04T12:41:06.3291772Z Finished distributed/_tools/test_memory_tracker 1/1 ... [2025-12-04 12:41:06.328536][2235890.785855686], took 0.10min
2025-12-04T12:41:06.3301183Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:41:06.3308456Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:41:06.3311242Z Running distributed/checkpoint/_experimental/test_builder 1/1 ... [2025-12-04 12:41:06.331000][2235890.788324467]
2025-12-04T12:41:06.3311718Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:41:06.3313409Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_builder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:41:06.331204]
2025-12-04T12:41:10.4520308Z 
2025-12-04T12:41:10.4521183Z distributed/checkpoint/_experimental/test_builder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_builder_1.1_47635a76130eb5df_.log
2025-12-04T12:41:10.4523392Z Running 4 items in this shard: test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_async_checkpointer, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer_with_config_first, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer_with_custom_config
2025-12-04T12:41:10.4524951Z 
2025-12-04T12:41:10.4525281Z Finished distributed/checkpoint/_experimental/test_builder 1/1 ... [2025-12-04 12:41:10.451794][2235894.909112865], took 0.07min
2025-12-04T12:41:10.4534464Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:41:10.4540670Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:41:10.4542721Z Running distributed/_composable/test_replicate_with_fsdp 1/1 ... [2025-12-04 12:41:10.454171][2235894.911494827]
2025-12-04T12:41:10.4543102Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:41:10.4545091Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/test_replicate_with_fsdp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:41:10.454370]
2025-12-04T12:41:39.8104813Z 
2025-12-04T12:41:39.8105585Z distributed/_composable/test_replicate_with_fsdp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_fsdp_1.1_1a335d4f323814e8_.log
2025-12-04T12:41:39.8106925Z Running 5 items in this shard: test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_tp_device_mesh, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_transformer, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_transformer_managed_modules, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_train_parity_2d_mlp, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_train_replicate_fsdp
2025-12-04T12:41:39.8107737Z 
2025-12-04T12:41:39.8107906Z Finished distributed/_composable/test_replicate_with_fsdp 1/1 ... [2025-12-04 12:41:39.810086][2235924.267407518], took 0.49min
2025-12-04T12:41:39.8109934Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:41:39.8117932Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:41:39.8119608Z Running distributed/test_backends 1/1 ... [2025-12-04 12:41:39.811805][2235924.26912956]
2025-12-04T12:41:39.8120707Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:41:39.8121536Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_backends.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:41:39.811970]
2025-12-04T12:41:42.6806169Z 
2025-12-04T12:41:42.6807277Z distributed/test_backends 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_backends_1.1_994755d2331f1e01_.log
2025-12-04T12:41:42.6809284Z Running 2 items in this shard: test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_create_pg_cuda, test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_device_to_backend_mapping_cuda
2025-12-04T12:41:42.6809999Z 
2025-12-04T12:41:42.6810265Z Finished distributed/test_backends 1/1 ... [2025-12-04 12:41:42.680134][2235927.13745633], took 0.05min
2025-12-04T12:41:42.6811293Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:41:42.6819245Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:41:42.6820072Z Running distributed/tensor/test_experimental_ops 1/1 ... [2025-12-04 12:41:42.681870][2235927.139194393]
2025-12-04T12:41:42.6820510Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:41:42.6823125Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:41:42.682041]
2025-12-04T12:41:58.0212280Z 
2025-12-04T12:41:58.0213193Z distributed/tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_experimental_ops_1.1_41b5facccfac6ad9_.log
2025-12-04T12:41:58.0214783Z Running 6 items in this shard: test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTestWithLocalTensor::test_bernoulli, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTestWithLocalTensor::test_nll, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTestWithLocalTensor::test_slice
2025-12-04T12:41:58.0215952Z 
2025-12-04T12:41:58.0216162Z Finished distributed/tensor/test_experimental_ops 1/1 ... [2025-12-04 12:41:58.020895][2235942.478217043], took 0.26min
2025-12-04T12:41:58.0220107Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:41:58.0226503Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:41:58.0229284Z Running distributed/checkpoint/test_async_process_executor 1/1 ... [2025-12-04 12:41:58.022836][2235942.480161052]
2025-12-04T12:41:58.0229576Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:41:58.0231146Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_async_process_executor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:41:58.023013]
2025-12-04T12:42:23.8796065Z 
2025-12-04T12:42:23.8797385Z distributed/checkpoint/test_async_process_executor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_async_process_executor_1.1_513157e0add6b790_.log
2025-12-04T12:42:23.8801193Z Running 5 items in this shard: test/distributed/checkpoint/test_async_process_executor.py::TestAsyncProcessExecutor::test_checkpoint_save_failure_continues_serving, test/distributed/checkpoint/test_async_process_executor.py::TestAsyncProcessExecutorPrefixStore::test_checkpoint_save_with_prefix_store_enabled, test/distributed/checkpoint/test_async_process_executor.py::TestProcessGroupInitInfo::test_process_group_init_info_with_default_pg, test/distributed/checkpoint/test_async_process_executor.py::TestProcessGroupInitInfo::test_process_group_init_info_with_prefix_store_env_var, test/distributed/checkpoint/test_async_process_executor.py::TestProcessGroupInitInfo::test_process_group_init_info_without_prefix_store_env_var
2025-12-04T12:42:23.8803746Z 
2025-12-04T12:42:23.8804108Z Finished distributed/checkpoint/test_async_process_executor 1/1 ... [2025-12-04 12:42:23.879478][2235968.336796026], took 0.43min
2025-12-04T12:42:23.8807175Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:42:23.8813727Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:42:23.8815923Z Running distributed/test_nvshmem 1/1 ... [2025-12-04 12:42:23.881519][2235968.338843543]
2025-12-04T12:42:23.8816231Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:42:23.8818658Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_nvshmem.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:42:23.881697]
2025-12-04T12:42:25.8995000Z 
2025-12-04T12:42:25.8995859Z distributed/test_nvshmem 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nvshmem_1.1_e4d6349b80bdc56c_.log
2025-12-04T12:42:25.9009593Z Running 47 items in this shard: test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_alloc, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_alloc_without_device_context, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_get_remote_tensor, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_get_remote_tensors, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_handle_offset, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_mempool_compute_ops, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_mempool_tensor_factory, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_mempool_tensor_w_collective, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_nvshmem_get, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_nvshmem_put, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev_2d_align_1, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev_2d_align_16, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev_2d_align_8, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev_2d_offset, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_nvshmem_all_to_all, test/distributed/test_nvshmem.py::DispatchCombineTest::test_dispatch_combine_align_1, test/distributed/test_nvshmem.py::DispatchCombineTest::test_dispatch_combine_align_16, test/distributed/test_nvshmem.py::DispatchCombineTest::test_dispatch_combine_align_8, test/distributed/test_nvshmem.py::DispatchCombineInSubgroups::test_dispatch_combine_subgroup, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_128_root_ratio_1_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_128_root_ratio_1_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_128_root_ratio_1_float32, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_128_root_ratio_2_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_128_root_ratio_2_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_128_root_ratio_2_float32, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_32_root_ratio_1_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_32_root_ratio_1_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_32_root_ratio_1_float32, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_32_root_ratio_2_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_32_root_ratio_2_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_32_root_ratio_2_float32, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_512_root_ratio_1_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_512_root_ratio_1_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_512_root_ratio_1_float32, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_512_root_ratio_2_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_512_root_ratio_2_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_multi_root_tile_reduce_tile_size_512_root_ratio_2_float32, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_128_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_128_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_128_float32, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_32_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_32_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_32_float32, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_512_bfloat16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_512_float16, test/distributed/test_nvshmem.py::NVSHMEMTileCommTest::test_tile_reduce_tile_size_512_float32
2025-12-04T12:42:25.9018869Z 
2025-12-04T12:42:25.9019002Z Finished distributed/test_nvshmem 1/1 ... [2025-12-04 12:42:25.899203][2235970.356522926], took 0.03min
2025-12-04T12:42:25.9019495Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:42:25.9019937Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:42:25.9020207Z Running distributed/tensor/test_attention 1/1 ... [2025-12-04 12:42:25.901579][2235970.358903548]
2025-12-04T12:42:25.9020432Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:42:25.9020876Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:42:25.901778]
2025-12-04T12:44:25.1030771Z 
2025-12-04T12:44:25.1031953Z distributed/tensor/test_attention 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_attention_1.1_557142b2c4616df3_.log
2025-12-04T12:44:25.1037477Z Running 14 items in this shard: test/distributed/tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa, test/distributed/tensor/test_attention.py::CPFlexAttentionTest::test_cp_flex_attention_causal_mask, test/distributed/tensor/test_attention.py::CPFlexAttentionTest::test_cp_flex_attention_document_mask, test/distributed/tensor/test_attention.py::TestCPCustomOps::test_flex_cp_custom_op, test/distributed/tensor/test_attention.py::TestSharding::test_attention_shard_without_cp, test/distributed/tensor/test_attention.py::TestSharding::test_context_parallel_shard, test/distributed/tensor/test_attention.py::RingAttentionTestWithLocalTensor::test_is_causal_behavior, test/distributed/tensor/test_attention.py::RingAttentionTestWithLocalTensor::test_ring_attention_sdpa, test/distributed/tensor/test_attention.py::CPFlexAttentionTestWithLocalTensor::test_cp_flex_attention_causal_mask, test/distributed/tensor/test_attention.py::CPFlexAttentionTestWithLocalTensor::test_cp_flex_attention_document_mask, test/distributed/tensor/test_attention.py::TestCPCustomOpsWithLocalTensor::test_flex_cp_custom_op, test/distributed/tensor/test_attention.py::TestShardingWithLocalTensor::test_attention_shard_without_cp, test/distributed/tensor/test_attention.py::TestShardingWithLocalTensor::test_context_parallel_shard
2025-12-04T12:44:25.1042006Z 
2025-12-04T12:44:25.1042254Z Finished distributed/tensor/test_attention 1/1 ... [2025-12-04 12:44:25.102780][2236089.560099034], took 1.99min
2025-12-04T12:44:25.1047140Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:44:25.1057080Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:44:25.1061191Z Running distributed/tensor/test_convolution_ops 1/1 ... [2025-12-04 12:44:25.105993][2236089.563309612]
2025-12-04T12:44:25.1061630Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:44:25.1063989Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:44:25.106275]
2025-12-04T12:45:51.4581850Z 
2025-12-04T12:45:51.4582815Z distributed/tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_convolution_ops_1.1_408d7d1cb46c072e_.log
2025-12-04T12:45:51.4587803Z Running 16 items in this shard: test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_conv1d, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_conv2d_module_no_bias, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_conv2d_no_bias_backward, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_conv2d_no_bias_compile, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_conv3d, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_conv_backward_none_grad_inp, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTestWithLocalTensor::test_conv1d, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTestWithLocalTensor::test_conv2d_module_no_bias, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTestWithLocalTensor::test_conv2d_no_bias_backward, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTestWithLocalTensor::test_conv2d_no_bias_compile, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTestWithLocalTensor::test_conv3d, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTestWithLocalTensor::test_conv_backward_none_grad_inp, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTestWithLocalTensor::test_depthwise_convolution, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTestWithLocalTensor::test_downsampling_convolution
2025-12-04T12:45:51.4591528Z 
2025-12-04T12:45:51.4591725Z Finished distributed/tensor/test_convolution_ops 1/1 ... [2025-12-04 12:45:51.457890][2236175.915209395], took 1.44min
2025-12-04T12:45:51.4592349Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:45:51.4599994Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:45:51.4603154Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2025-12-04 12:45:51.460231][2236175.917555117]
2025-12-04T12:45:51.4603404Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:45:51.4605263Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:45:51.460421]
2025-12-04T12:46:02.3916646Z 
2025-12-04T12:46:02.3918336Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_ce4eb7595654c63d_.log
2025-12-04T12:46:02.3919850Z Running 2 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_assert_same_keys, test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect
2025-12-04T12:46:02.3920645Z 
2025-12-04T12:46:02.3920992Z Finished distributed/checkpoint/test_save_load_api 1/1 ... [2025-12-04 12:46:02.391279][2236186.848598847], took 0.18min
2025-12-04T12:46:02.3928497Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:46:02.3936441Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:46:02.3939404Z Running distributed/test_debug 1/1 ... [2025-12-04 12:46:02.393836][2236186.851159896]
2025-12-04T12:46:02.3939739Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:46:02.3941747Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_debug.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:46:02.394030]
2025-12-04T12:46:04.6625525Z 
2025-12-04T12:46:04.6627067Z distributed/test_debug 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_debug_1.1_c3e8e4478b5bd9c0_.log
2025-12-04T12:46:04.6627972Z Running 1 items in this shard: test/distributed/test_debug.py::TestDebug::test_all
2025-12-04T12:46:04.6628295Z 
2025-12-04T12:46:04.6628605Z Finished distributed/test_debug 1/1 ... [2025-12-04 12:46:04.662139][2236189.119458652], took 0.04min
2025-12-04T12:46:04.6632714Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:46:04.6639583Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:46:04.6642640Z Running distributed/test_overlap_bucketing_unit 1/1 ... [2025-12-04 12:46:04.664173][2236189.12149732]
2025-12-04T12:46:04.6643013Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:46:04.6644908Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_overlap_bucketing_unit.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:46:04.664364]
2025-12-04T12:46:10.7882611Z 
2025-12-04T12:46:10.7883776Z distributed/test_overlap_bucketing_unit 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_overlap_bucketing_unit_1.1_a0c28a5ce0c5e24f_.log
2025-12-04T12:46:10.7888447Z Running 9 items in this shard: test/distributed/test_overlap_bucketing_unit.py::TestOverlapPreservingBucketing::test_can_bucket_all_reduce, test/distributed/test_overlap_bucketing_unit.py::TestOverlapPreservingBucketing::test_can_bucket_independent_collectives, test/distributed/test_overlap_bucketing_unit.py::TestOverlapPreservingBucketing::test_can_bucket_multidtype_collectives, test/distributed/test_overlap_bucketing_unit.py::TestOverlapPreservingBucketing::test_can_bucket_with_convert_dtype_as_hiding_nodes, test/distributed/test_overlap_bucketing_unit.py::TestOverlapPreservingBucketing::test_can_bucket_with_multiple_hiding_nodes, test/distributed/test_overlap_bucketing_unit.py::TestOverlapPreservingBucketing::test_cant_bucket_ag_with_rs_hiding_interval_between_final_mm_hidden_False, test/distributed/test_overlap_bucketing_unit.py::TestOverlapPreservingBucketing::test_cant_bucket_ag_with_rs_hiding_interval_between_final_mm_hidden_True, test/distributed/test_overlap_bucketing_unit.py::TestOverlapPreservingBucketing::test_cant_bucket_nested_hiding_intervals, test/distributed/test_overlap_bucketing_unit.py::TestCrossPGOverlap::test_cross_pg_prefetch_during_exposed_wait
2025-12-04T12:46:10.7892330Z 
2025-12-04T12:46:10.7892630Z Finished distributed/test_overlap_bucketing_unit 1/1 ... [2025-12-04 12:46:10.787917][2236195.245236382], took 0.10min
2025-12-04T12:46:10.7894286Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:46:10.7902641Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:46:10.7906512Z Running distributed/checkpoint/_experimental/test_checkpoint_writer 1/1 ... [2025-12-04 12:46:10.790518][2236195.247841881]
2025-12-04T12:46:10.7907128Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:46:10.7908654Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_checkpoint_writer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:46:10.790725]
2025-12-04T12:46:12.9087027Z 
2025-12-04T12:46:12.9088318Z distributed/checkpoint/_experimental/test_checkpoint_writer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_checkpoint_writer_1.1_1908922172db3255_.log
2025-12-04T12:46:12.9092245Z Running 8 items in this shard: test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriterConfig::test_custom_values, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriterConfig::test_default_values, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_close, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_calls_barrier, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_calls_commit_hooks, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_creates_checkpoint_file, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_without_barrier, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_without_commit_hook
2025-12-04T12:46:12.9095308Z 
2025-12-04T12:46:12.9095692Z Finished distributed/checkpoint/_experimental/test_checkpoint_writer 1/1 ... [2025-12-04 12:46:12.908360][2236197.365679812], took 0.04min
2025-12-04T12:46:12.9099821Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:46:12.9108798Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:46:12.9111592Z Running distributed/optim/test_named_optimizer 1/1 ... [2025-12-04 12:46:12.911062][2236197.368386579]
2025-12-04T12:46:12.9111881Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:46:12.9113847Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:46:12.911260]
2025-12-04T12:46:14.1616963Z 
2025-12-04T12:46:14.1618132Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_03ad6cda5b372ee5_.log
2025-12-04T12:46:14.1618825Z 
2025-12-04T12:46:14.1619178Z Finished distributed/optim/test_named_optimizer 1/1 ... [2025-12-04 12:46:14.161319][2236198.618638528], took 0.02min
2025-12-04T12:46:14.1629321Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:46:14.1637616Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:46:14.1640767Z Running distributed/checkpoint/_experimental/test_checkpointer 1/1 ... [2025-12-04 12:46:14.163975][2236198.621299625]
2025-12-04T12:46:14.1641188Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:46:14.1643103Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_checkpointer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:46:14.164173]
2025-12-04T12:46:35.0600011Z 
2025-12-04T12:46:35.0601606Z distributed/checkpoint/_experimental/test_checkpointer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_checkpointer_1.1_2d7db23dd7a8e22a_.log
2025-12-04T12:46:35.0607881Z Running 11 items in this shard: test/distributed/checkpoint/_experimental/test_checkpointer.py::TestCheckpointer::test_load_strict_mode, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestCheckpointer::test_load_with_map_location, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestCheckpointer::test_nested_dict_partial_load, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestCheckpointer::test_partial_load, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestCheckpointer::test_save_and_load_basic, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestCheckpointer::test_save_with_kwargs, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestAsyncCheckpointerSpecific::test_async_error_handling, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestAsyncCheckpointerSpecific::test_async_future_results, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestAsyncCheckpointerSpecific::test_async_multiple_saves_ordering, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestAsyncCheckpointerSpecific::test_async_returns_futures, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestAsyncCheckpointerSpecific::test_async_sequential_saves_wait
2025-12-04T12:46:35.0611945Z 
2025-12-04T12:46:35.0612287Z Finished distributed/checkpoint/_experimental/test_checkpointer 1/1 ... [2025-12-04 12:46:35.059628][2236219.516947966], took 0.35min
2025-12-04T12:46:35.0613949Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:46:35.0620387Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:46:35.0623997Z Running distributed/tensor/test_api 1/1 ... [2025-12-04 12:46:35.062221][2236219.519545834]
2025-12-04T12:46:35.0624447Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:46:35.0625591Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:46:35.062421]
2025-12-04T12:47:28.7089149Z 
2025-12-04T12:47:28.7090379Z distributed/tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_api_1.1_d7aac6a400ae40a9_.log
2025-12-04T12:47:28.7095248Z Running 18 items in this shard: test/distributed/tensor/test_api.py::DTensorAPITest::test_checkpoint_apis_check_partial_placement, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_tensor_rank, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_checkpoint_apis_check_partial_placement, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_distribute_module, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_distribute_module_casting, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_distribute_module_input_fn_output_fn, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_distribute_module_input_fn_output_fn_warning, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_distribute_module_meta, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_distribute_tensor_errors, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_distribute_tensor_rank, test/distributed/tensor/test_api.py::DTensorAPITestWithLocalTensor::test_distribute_tensor_uneven_sharding
2025-12-04T12:47:28.7098453Z 
2025-12-04T12:47:28.7098603Z Finished distributed/tensor/test_api 1/1 ... [2025-12-04 12:47:28.708633][2236273.165953037], took 0.89min
2025-12-04T12:47:28.7099123Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:47:28.7104572Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:47:28.7107037Z Running distributed/checkpoint/test_traverse 1/1 ... [2025-12-04 12:47:28.710630][2236273.167954675]
2025-12-04T12:47:28.7107250Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:47:28.7109213Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:47:28.710801]
2025-12-04T12:47:30.8289423Z 
2025-12-04T12:47:30.8290483Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_08cd864abe5f883a_.log
2025-12-04T12:47:30.8294203Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict
2025-12-04T12:47:30.8296619Z 
2025-12-04T12:47:30.8297176Z Finished distributed/checkpoint/test_traverse 1/1 ... [2025-12-04 12:47:30.828588][2236275.285908713], took 0.04min
2025-12-04T12:47:30.8303693Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:47:30.8310922Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:47:30.8313228Z Running distributed/tensor/test_random_ops 1/1 ... [2025-12-04 12:47:30.831232][2236275.288556531]
2025-12-04T12:47:30.8313575Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:47:30.8315279Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:47:30.831405]
2025-12-04T12:48:32.4410025Z 
2025-12-04T12:48:32.4411041Z distributed/tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_random_ops_1.1_765e2799b3d774d5_.log
2025-12-04T12:48:32.4420688Z Running 28 items in this shard: test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_fsdp_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_with_user_generator, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_meta_tensor_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed_submesh, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_philox_state_seed_roundtrip, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_pipeline_parallel_manual_seed, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpsTest3D::test_hsdp_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTestWithLocalTensor::test_fsdp_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTestWithLocalTensor::test_init_ops, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTestWithLocalTensor::test_init_with_user_generator, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTestWithLocalTensor::test_meta_tensor_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTestWithLocalTensor::test_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTestWithLocalTensor::test_deterministic_dropout_1d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTestWithLocalTensor::test_deterministic_rand_1d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTestWithLocalTensor::test_deterministic_uniform_2d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTestWithLocalTensor::test_manual_seed, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTestWithLocalTensor::test_manual_seed_submesh, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTestWithLocalTensor::test_philox_state_seed_roundtrip, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTestWithLocalTensor::test_pipeline_parallel_manual_seed, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTestWithLocalTensor::test_rng_tracker_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpsTest3DWithLocalTensor::test_hsdp_tp_model_meta_init
2025-12-04T12:48:32.4428118Z 
2025-12-04T12:48:32.4428298Z Finished distributed/tensor/test_random_ops 1/1 ... [2025-12-04 12:48:32.440751][2236336.898072145], took 1.03min
2025-12-04T12:48:32.4429061Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:48:32.4429593Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:48:32.4432151Z Running distributed/checkpoint/_experimental/test_staging 1/1 ... [2025-12-04 12:48:32.443113][2236336.900437827]
2025-12-04T12:48:32.4432462Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:48:32.4434272Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_staging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:48:32.443288]
2025-12-04T12:48:34.8614360Z 
2025-12-04T12:48:34.8615689Z distributed/checkpoint/_experimental/test_staging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_staging_1.1_689fc7f7bc647c69_.log
2025-12-04T12:48:34.8619969Z Running 7 items in this shard: test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_async_staging, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_cuda_non_blocking_without_cuda, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_cuda_tensors_staging, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_different_option_combinations, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_multiple_staging_operations, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_resource_cleanup, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_sync_staging
2025-12-04T12:48:34.8623123Z 
2025-12-04T12:48:34.8623573Z Finished distributed/checkpoint/_experimental/test_staging 1/1 ... [2025-12-04 12:48:34.861108][2236339.318429008], took 0.04min
2025-12-04T12:48:34.8627311Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:48:34.8634575Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:48:34.8634950Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-12-04T12:48:34.8635237Z Uploading artifacts took 0.00 seconds
2025-12-04T12:48:34.8638853Z Running distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2025-12-04 12:48:34.863617][2236339.320941687]
2025-12-04T12:48:34.8639509Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:48:34.8640587Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:48:34.863792]
2025-12-04T12:48:45.2446649Z 
2025-12-04T12:48:45.2448191Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_48e6c15cbcfb47a0_.log
2025-12-04T12:48:45.2450467Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py::TestFsdpTpCheckpointConversion::test_fsdp_to_tp
2025-12-04T12:48:45.2451134Z 
2025-12-04T12:48:45.2451595Z Finished distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2025-12-04 12:48:45.244289][2236349.701610413], took 0.17min
2025-12-04T12:48:45.2458314Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:48:45.2461898Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:48:45.2465334Z Running distributed/launcher/test_api 1/1 ... [2025-12-04 12:48:45.246396][2236349.703720609]
2025-12-04T12:48:45.2465691Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:48:45.2467330Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/launcher/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:48:45.246585]
2025-12-04T12:48:47.4145427Z 
2025-12-04T12:48:47.4146570Z distributed/launcher/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.launcher.test_api_1.1_33a6969a3af2f7fc_.log
2025-12-04T12:48:47.4148282Z Running 2 items in this shard: test/distributed/launcher/test_api.py::LauncherApiTest::test_launch_agent_default_signals, test/distributed/launcher/test_api.py::LauncherApiTest::test_launch_agent_sets_signals_env_var
2025-12-04T12:48:47.4149185Z 
2025-12-04T12:48:47.4149519Z Finished distributed/launcher/test_api 1/1 ... [2025-12-04 12:48:47.414147][2236351.871465747], took 0.04min
2025-12-04T12:48:47.4157755Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:48:47.4165676Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:48:47.4168932Z Running distributed/elastic/multiprocessing/test_api 1/1 ... [2025-12-04 12:48:47.416753][2236351.874077195]
2025-12-04T12:48:47.4169372Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:48:47.4171162Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/multiprocessing/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:48:47.416944]
2025-12-04T12:48:49.5346632Z 
2025-12-04T12:48:49.5348151Z distributed/elastic/multiprocessing/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.test_api_1.1_35200d98d406b7c6_.log
2025-12-04T12:48:49.5352391Z Running 7 items in this shard: test/distributed/elastic/multiprocessing/test_api.py::SignalHandlingTest::test_start_handles_invalid_signals, test/distributed/elastic/multiprocessing/test_api.py::SignalHandlingTest::test_start_handles_windows_signals, test/distributed/elastic/multiprocessing/test_api.py::SignalHandlingTest::test_start_not_main_thread, test/distributed/elastic/multiprocessing/test_api.py::SignalHandlingTest::test_start_registers_custom_signals, test/distributed/elastic/multiprocessing/test_api.py::SignalHandlingTest::test_start_registers_default_signals, test/distributed/elastic/multiprocessing/test_api.py::SignalHandlingTest::test_start_supports_sigusr1_and_sigusr2, test/distributed/elastic/multiprocessing/test_api.py::SignalHandlingTest::test_terminate_process_handler
2025-12-04T12:48:49.5355724Z 
2025-12-04T12:48:49.5356158Z Finished distributed/elastic/multiprocessing/test_api 1/1 ... [2025-12-04 12:48:49.534285][2236353.99160331], took 0.04min
2025-12-04T12:48:49.5362405Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:48:49.5369572Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:48:49.5372530Z Running distributed/fsdp/test_shard_utils 1/1 ... [2025-12-04 12:48:49.537152][2236353.994476943]
2025-12-04T12:48:49.5374202Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:48:49.5374788Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:48:49.537343]
2025-12-04T12:48:59.9172731Z 
2025-12-04T12:48:59.9173655Z distributed/fsdp/test_shard_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_shard_utils_1.1_518111ca11a62a1c_.log
2025-12-04T12:48:59.9174948Z Running 2 items in this shard: test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributed::test_create_chunk_sharded_tensor, test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributedDTensor::test_create_chunk_dtensor
2025-12-04T12:48:59.9175662Z 
2025-12-04T12:48:59.9175935Z Finished distributed/fsdp/test_shard_utils 1/1 ... [2025-12-04 12:48:59.916872][2236364.374191633], took 0.17min
2025-12-04T12:48:59.9187420Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:48:59.9197507Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:48:59.9199495Z Running distributed/tensor/experimental/test_local_map 1/1 ... [2025-12-04 12:48:59.919787][2236364.377111287]
2025-12-04T12:48:59.9199847Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:48:59.9201201Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/experimental/test_local_map.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:48:59.919979]
2025-12-04T12:49:36.1412101Z 
2025-12-04T12:49:36.1413553Z distributed/tensor/experimental/test_local_map 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.experimental.test_local_map_1.1_5a64b13078f31cb5_.log
2025-12-04T12:49:36.1417249Z Running 6 items in this shard: test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_correctness, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_in_placements, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_out_placements, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_redistribute, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_with_grad_placement, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_multi_mesh_inputs
2025-12-04T12:49:36.1420021Z 
2025-12-04T12:49:36.1420459Z Finished distributed/tensor/experimental/test_local_map 1/1 ... [2025-12-04 12:49:36.140795][2236400.598114805], took 0.60min
2025-12-04T12:49:36.1425857Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:49:36.1432609Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:49:36.1438128Z Running distributed/test_local_tensor 1/1 ... [2025-12-04 12:49:36.143426][2236400.600750612]
2025-12-04T12:49:36.1438581Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:49:36.1439902Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_local_tensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:49:36.143602]
2025-12-04T12:49:38.5619929Z 
2025-12-04T12:49:38.5621044Z distributed/test_local_tensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_local_tensor_1.1_583349147849c60b_.log
2025-12-04T12:49:38.5627397Z Running 21 items in this shard: test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_basic_arithmetic_operations, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_collectives_within_local_tensor_mode, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_empty_local_tensors, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_even_sharding_mean_is_partial, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_local_tensor_creation_fails_with_grad_tensors, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_local_tensor_dtype_consistency, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_local_tensor_mode, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_mixed_operations_with_regular_tensors, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_scalar_mul_reduction_bug, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_uneven_sharding_mean_bug, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_uneven_sharding_prod, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_all_gather_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_all_gather_into_tensor_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_all_reduce_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_all_to_all_single_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_broadcast_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_collective_reduction_operations, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_reduce_scatter_tensor_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld4::test_dtensor_cat, test/distributed/test_local_tensor.py::TestLocalTensorWorld8::test_dtensor_addmm, test/distributed/test_local_tensor.py::TestLocalRunner::test_dp_pp
2025-12-04T12:49:38.5633217Z 
2025-12-04T12:49:38.5633401Z Finished distributed/test_local_tensor 1/1 ... [2025-12-04 12:49:38.561632][2236403.018951951], took 0.04min
2025-12-04T12:49:38.5635375Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:49:38.5641868Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:49:38.5644371Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2025-12-04 12:49:38.564354][2236403.021678817]
2025-12-04T12:49:38.5644657Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:49:38.5646255Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:49:38.564521]
2025-12-04T12:49:40.9329065Z 
2025-12-04T12:49:40.9330101Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_ff597d50d70e0487_.log
2025-12-04T12:49:40.9332912Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls
2025-12-04T12:49:40.9334514Z 
2025-12-04T12:49:40.9334817Z Finished distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2025-12-04 12:49:40.932542][2236405.389863389], took 0.04min
2025-12-04T12:49:40.9337929Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:49:40.9344089Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:49:40.9346799Z Running distributed/checkpoint/test_tp_checkpoint 1/1 ... [2025-12-04 12:49:40.934589][2236405.391913336]
2025-12-04T12:49:40.9347148Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:49:40.9349006Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_tp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:49:40.934769]
2025-12-04T12:49:56.8258343Z 
2025-12-04T12:49:56.8259372Z distributed/checkpoint/test_tp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_ddbce705ef4cd483_.log
2025-12-04T12:49:56.8260685Z Running 2 items in this shard: test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint, test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint_load_on_meta_device
2025-12-04T12:49:56.8261563Z 
2025-12-04T12:49:56.8261857Z Finished distributed/checkpoint/test_tp_checkpoint 1/1 ... [2025-12-04 12:49:56.825513][2236421.282834325], took 0.26min
2025-12-04T12:49:56.8270887Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:49:56.8277811Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:49:56.8280323Z Running distributed/tensor/test_optimizers 1/1 ... [2025-12-04 12:49:56.827951][2236421.285275636]
2025-12-04T12:49:56.8280684Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:49:56.8282413Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/tensor/test_optimizers.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:49:56.828129]
2025-12-04T12:51:59.9121218Z 
2025-12-04T12:51:59.9122217Z distributed/tensor/test_optimizers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_optimizers_1.1_1997712158165579_.log
2025-12-04T12:51:59.9130578Z Running 24 items in this shard: test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_RMSprop_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adadelta_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adagrad_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adamax_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adamw_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_admaw_fused_across_meshes, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_asgd_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_nadam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_optimizer_foreach_supported_types_include_DTensor, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_radam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_sgd_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_RMSprop_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_adadelta_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_adagrad_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_adam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_adamax_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_adamw_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_admaw_fused_across_meshes, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_asgd_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_nadam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_optimizer_foreach_supported_types_include_DTensor, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_radam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizerWithLocalTensor::test_sgd_1d_sharding
2025-12-04T12:51:59.9136894Z 
2025-12-04T12:51:59.9137097Z Finished distributed/tensor/test_optimizers 1/1 ... [2025-12-04 12:51:59.911766][2236544.369086909], took 2.05min
2025-12-04T12:51:59.9137776Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:51:59.9141412Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:51:59.9143678Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2025-12-04 12:51:59.914278][2236544.371603178]
2025-12-04T12:51:59.9143947Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:51:59.9145660Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:51:59.914451]
2025-12-04T12:53:14.7981965Z 
2025-12-04T12:53:14.7983303Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_45ba590d1f45d701_.log
2025-12-04T12:53:14.7989503Z Running 10 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions0, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions1, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions2, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_with_uneven_shards, test/distributed/checkpoint/test_dtensor_resharding.py::TestCheckpointableReshard::test_uneven_reshard_with_checkpointable_api, test/distributed/checkpoint/test_dtensor_resharding.py::TestCheckpointableReshard::test_uneven_reshard_with_dtensor_shards_wrapper_api
2025-12-04T12:53:14.7993300Z 
2025-12-04T12:53:14.7994160Z Finished distributed/checkpoint/test_dtensor_resharding 1/1 ... [2025-12-04 12:53:14.797900][2236619.255220315], took 1.25min
2025-12-04T12:53:14.7996410Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:53:14.8001282Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:53:14.8003885Z Running distributed/test_symmetric_memory 1/1 ... [2025-12-04 12:53:14.800271][2236619.257596137]
2025-12-04T12:53:14.8004733Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:53:14.8005856Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_symmetric_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:53:14.800439]
2025-12-04T12:53:49.9209655Z 
2025-12-04T12:53:49.9210685Z distributed/test_symmetric_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_symmetric_memory_1.1_c572e3e86bd301f7_.log
2025-12-04T12:53:49.9235955Z Running 96 items in this shard: test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_allow_overlapping_devices, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_cuda_nvlink_connectivity_detection, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_get_backend, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_get_signal_pad, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_has_multicast_support, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_large_alloc, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_subgroup, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_matmul_gather_dim_2, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_False, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_True, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_False, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_True, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_matmul_reduce_scatter_scatter_dim_2, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_False, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_True, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_False, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_multimem_all_gather_matmul, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_optimal_layout_dim_0, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_optimal_layout_dim_1, test/distributed/test_symmetric_memory.py::AsyncTPTest::test_optimal_layout_dim_2, test/distributed/test_symmetric_memory.py::SymmMemEmptySetDeviceTest::test_empty_strided_p2p_persistent_set_device_False, test/distributed/test_symmetric_memory.py::SymmMemEmptySetDeviceTest::test_empty_strided_p2p_persistent_set_device_True, test/distributed/test_symmetric_memory.py::SymmMemEmptySetDeviceTest::test_empty_strided_p2p_set_device_False, test/distributed/test_symmetric_memory.py::SymmMemEmptySetDeviceTest::test_empty_strided_p2p_set_device_True, test/distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_barrier_timeout, test/distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_put_signal_timeout, test/distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_wait_signal_timeout, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_16, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_8, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_reduce_out_bfloat16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_reduce_out_bfloat16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_reduce_out_bfloat16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_reduce_out_float32_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_reduce_out_float32_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_reduce_out_float32_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_one_shot_all_reduce, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_reduce_scatter, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_reduce_scatter_corner_cases, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_two_shot_all_reduce, test/distributed/test_symmetric_memory.py::LoweringTest::test_lowering_one_shot_all_reduce, test/distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_memset32, test/distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_stream_write_value32
2025-12-04T12:53:49.9251617Z 
2025-12-04T12:53:49.9251747Z Finished distributed/test_symmetric_memory 1/1 ... [2025-12-04 12:53:49.920581][2236654.377903361], took 0.59min
2025-12-04T12:53:49.9252255Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:53:49.9252651Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:53:49.9252917Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2025-12-04 12:53:49.922592][2236654.379916518]
2025-12-04T12:53:49.9253141Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:53:49.9253573Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:53:49.922773]
2025-12-04T12:55:46.4977601Z 
2025-12-04T12:55:46.4978791Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_8f4775956ad057b8_.log
2025-12-04T12:55:46.4982433Z Running 10 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu_ac, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp
2025-12-04T12:55:46.4985183Z 
2025-12-04T12:55:46.4985512Z Finished distributed/_composable/test_replicate_with_compiler 1/1 ... [2025-12-04 12:55:46.497419][2236770.954736734], took 1.94min
2025-12-04T12:55:46.4994501Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:55:46.5001362Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:55:46.5004000Z Running distributed/_composable/fsdp/test_fully_shard_autograd 1/1 ... [2025-12-04 12:55:46.500297][2236770.957621387]
2025-12-04T12:55:46.5004340Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:55:46.5006031Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_autograd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:55:46.500474]
2025-12-04T12:56:25.1755030Z 
2025-12-04T12:56:25.1757102Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_autograd_1.1_1c5aadaae39ee0d1_.log
2025-12-04T12:56:25.1759904Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_nontensor_activations, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_module, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_output, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardPostAccGradHookMultiThread::test_post_acc_grad_hook_runs, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardPostAccGradHookMultiProcess::test_post_acc_grad_hook_optim_parity
2025-12-04T12:56:25.1762158Z 
2025-12-04T12:56:25.1762487Z Finished distributed/_composable/fsdp/test_fully_shard_autograd 1/1 ... [2025-12-04 12:56:25.175173][2236809.632492922], took 0.64min
2025-12-04T12:56:25.1771376Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:56:25.1778510Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:56:25.1781012Z Running distributed/_composable/test_composability/test_2d_composability 1/1 ... [2025-12-04 12:56:25.178000][2236809.635324826]
2025-12-04T12:56:25.1781407Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:56:25.1782973Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/test_composability/test_2d_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:56:25.178174]
2025-12-04T12:58:54.3386873Z 
2025-12-04T12:58:54.3388199Z distributed/_composable/test_composability/test_2d_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_2d_composability_1.1_c168312368cfb4b5_.log
2025-12-04T12:58:54.3397340Z Running 18 items in this shard: test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_tp_with_fsdp_offloading, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_mlp, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DStateDict::test_fully_shard_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::Test2dFSDP1ParallelIntegration::test_2d_ddp_integration_functionality, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp1_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp_2d_extension
2025-12-04T12:58:54.3403792Z 
2025-12-04T12:58:54.3404060Z Finished distributed/_composable/test_composability/test_2d_composability 1/1 ... [2025-12-04 12:58:54.338528][2236958.795847413], took 2.49min
2025-12-04T12:58:54.3404799Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T12:58:54.3412012Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T12:58:54.3414458Z Running distributed/fsdp/test_fsdp_optim_state 1/1 ... [2025-12-04 12:58:54.341355][2236958.798679247]
2025-12-04T12:58:54.3414712Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T12:58:54.3416792Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 12:58:54.341543]
2025-12-04T13:06:49.0549652Z 
2025-12-04T13:06:49.0550861Z distributed/fsdp/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_9c295fc20ca7f32e_.log
2025-12-04T13:06:49.0576810Z Running 60 items in this shard: test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_compatible_with_trec, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_nested, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_keys, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_nested_invalid, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_interface_arguments, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_no_grad, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_input_warning, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_without_param_groups, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_names, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type0, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type1, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_state_dict_with_none_tensor_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_use_orig_params, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_empty_optimizer_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_no_shard
2025-12-04T13:06:49.0590431Z 
2025-12-04T13:06:49.0590574Z Finished distributed/fsdp/test_fsdp_optim_state 1/1 ... [2025-12-04 13:06:49.056001][2237433.513322063], took 7.91min
2025-12-04T13:06:49.0591037Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:06:49.0591443Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:06:49.0591687Z Running distributed/fsdp/test_fsdp_state_dict 2/2 ... [2025-12-04 13:06:49.058286][2237433.515610696]
2025-12-04T13:06:49.0591891Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:06:49.0592349Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:06:49.058475]
2025-12-04T13:14:41.7989237Z 
2025-12-04T13:14:41.7990307Z distributed/fsdp/test_fsdp_state_dict 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_c4b3dba79d915f2f_.log
2025-12-04T13:14:41.8028774Z Running 101 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_sharded_load_multi_backend_pg, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config
2025-12-04T13:14:41.8054673Z 
2025-12-04T13:14:41.8054814Z Finished distributed/fsdp/test_fsdp_state_dict 2/2 ... [2025-12-04 13:14:41.799510][2237906.256831649], took 7.88min
2025-12-04T13:14:41.8055267Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:14:41.8055670Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:14:41.8055939Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-12-04T13:14:41.8056119Z Uploading artifacts took 0.00 seconds
2025-12-04T13:14:41.8056315Z Running distributed/test_distributed_spawn 2/7 ... [2025-12-04 13:14:41.801522][2237906.258846906]
2025-12-04T13:14:41.8056540Z MPI not available -- MPI backend tests will be skipped
2025-12-04T13:14:41.8056732Z Running distributed tests for the test backend with env init_method
2025-12-04T13:14:41.8056942Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:14:41.8057365Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:14:41.801935]
2025-12-04T13:14:43.7673185Z 
2025-12-04T13:14:43.7674271Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_c46236630bc975f8_.log
2025-12-04T13:14:43.7675147Z Running 0 items in this shard:
2025-12-04T13:14:43.7675341Z 
2025-12-04T13:14:43.7679150Z Running distributed tests for the test backend with file init_method
2025-12-04T13:14:43.7679758Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:14:43.7682681Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:14:43.768103]
2025-12-04T13:14:45.7095330Z 
2025-12-04T13:14:45.7096462Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_ab1754a38f8308e9_.log
2025-12-04T13:14:45.7097646Z Running 0 items in this shard:
2025-12-04T13:14:45.7097863Z 
2025-12-04T13:14:45.7100812Z Running distributed tests for the nccl backend with env init_method
2025-12-04T13:14:45.7101399Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:14:45.7103832Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:14:45.710221]
2025-12-04T13:17:42.1980023Z 
2025-12-04T13:17:42.1981039Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_bf42df857af74722_.log
2025-12-04T13:17:42.1994946Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-12-04T13:17:42.2004595Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient
2025-12-04T13:17:42.2005213Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine
2025-12-04T13:17:42.2005842Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2025-12-04T13:17:42.2006399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad
2025-12-04T13:17:42.2006965Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2025-12-04T13:17:42.2007473Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max
2025-12-04T13:17:42.2008048Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum
2025-12-04T13:17:42.2008553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max
2025-12-04T13:17:42.2009038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max
2025-12-04T13:17:42.2009510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max
2025-12-04T13:17:42.2009886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product
2025-12-04T13:17:42.2010267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max
2025-12-04T13:17:42.2010627Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min
2025-12-04T13:17:42.2010977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum
2025-12-04T13:17:42.2011344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async
2025-12-04T13:17:42.2011752Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2025-12-04T13:17:42.2012186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2025-12-04T13:17:42.2012590Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters
2025-12-04T13:17:42.2012937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier
2025-12-04T13:17:42.2013289Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2025-12-04T13:17:42.2013652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast
2025-12-04T13:17:42.2014026Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks
2025-12-04T13:17:42.2014429Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks
2025-12-04T13:17:42.2014800Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2025-12-04T13:17:42.2015147Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank
2025-12-04T13:17:42.2015507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2025-12-04T13:17:42.2015868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv
2025-12-04T13:17:42.2016270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size
2025-12-04T13:17:42.2016738Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager
2025-12-04T13:17:42.2017178Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group
2025-12-04T13:17:42.2017577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum
2025-12-04T13:17:42.2017947Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice
2025-12-04T13:17:42.2018339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks
2025-12-04T13:17:42.2018696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex
2025-12-04T13:17:42.2019050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2025-12-04T13:17:42.2019397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list
2025-12-04T13:17:42.2019748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2025-12-04T13:17:42.2020119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2025-12-04T13:17:42.2020488Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag
2025-12-04T13:17:42.2020864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler
2025-12-04T13:17:42.2021259Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-12-04T13:17:42.2021477Z 
2025-12-04T13:17:42.2021567Z Running distributed tests for the nccl backend with file init_method
2025-12-04T13:17:42.2021741Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:17:42.2022168Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:17:42.199244]
2025-12-04T13:20:38.3449949Z 
2025-12-04T13:20:38.3451137Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_86e1f58d080d7e81_.log
2025-12-04T13:20:38.3465210Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-12-04T13:20:38.3474677Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient
2025-12-04T13:20:38.3475246Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine
2025-12-04T13:20:38.3475774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2025-12-04T13:20:38.3476285Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad
2025-12-04T13:20:38.3476811Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2025-12-04T13:20:38.3477284Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max
2025-12-04T13:20:38.3477823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum
2025-12-04T13:20:38.3478292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max
2025-12-04T13:20:38.3478768Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max
2025-12-04T13:20:38.3479218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max
2025-12-04T13:20:38.3479709Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product
2025-12-04T13:20:38.3480151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max
2025-12-04T13:20:38.3480562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min
2025-12-04T13:20:38.3480975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum
2025-12-04T13:20:38.3481412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async
2025-12-04T13:20:38.3481891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2025-12-04T13:20:38.3482403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2025-12-04T13:20:38.3482836Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters
2025-12-04T13:20:38.3483167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier
2025-12-04T13:20:38.3483504Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2025-12-04T13:20:38.3483843Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast
2025-12-04T13:20:38.3484203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks
2025-12-04T13:20:38.3484582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks
2025-12-04T13:20:38.3484934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2025-12-04T13:20:38.3485258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank
2025-12-04T13:20:38.3485596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2025-12-04T13:20:38.3485934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv
2025-12-04T13:20:38.3486309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size
2025-12-04T13:20:38.3486707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager
2025-12-04T13:20:38.3487122Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group
2025-12-04T13:20:38.3487499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum
2025-12-04T13:20:38.3487852Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice
2025-12-04T13:20:38.3488228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks
2025-12-04T13:20:38.3488569Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex
2025-12-04T13:20:38.3488908Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2025-12-04T13:20:38.3489248Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list
2025-12-04T13:20:38.3489627Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2025-12-04T13:20:38.3489997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2025-12-04T13:20:38.3490384Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag
2025-12-04T13:20:38.3490754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler
2025-12-04T13:20:38.3491146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-12-04T13:20:38.3491359Z 
2025-12-04T13:20:38.3491450Z Running distributed tests for the gloo backend with env init_method
2025-12-04T13:20:38.3491620Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:20:38.3492053Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:20:38.346144]
2025-12-04T13:23:55.3978139Z 
2025-12-04T13:23:55.3979075Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_740f42b4221e8803_.log
2025-12-04T13:23:55.3990772Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-12-04T13:23:55.3999833Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient
2025-12-04T13:23:55.4000380Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine
2025-12-04T13:23:55.4000895Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2025-12-04T13:23:55.4001388Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad
2025-12-04T13:23:55.4001851Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2025-12-04T13:23:55.4002309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max
2025-12-04T13:23:55.4002774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum
2025-12-04T13:23:55.4003223Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max
2025-12-04T13:23:55.4003661Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max
2025-12-04T13:23:55.4004089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max
2025-12-04T13:23:55.4004593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product
2025-12-04T13:23:55.4005004Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max
2025-12-04T13:23:55.4005397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min
2025-12-04T13:23:55.4005788Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum
2025-12-04T13:23:55.4006266Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async
2025-12-04T13:23:55.4006722Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2025-12-04T13:23:55.4007253Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2025-12-04T13:23:55.4007702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters
2025-12-04T13:23:55.4008041Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier
2025-12-04T13:23:55.4008383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2025-12-04T13:23:55.4008728Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast
2025-12-04T13:23:55.4009101Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks
2025-12-04T13:23:55.4009489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks
2025-12-04T13:23:55.4009850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2025-12-04T13:23:55.4010177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank
2025-12-04T13:23:55.4010522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2025-12-04T13:23:55.4010972Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv
2025-12-04T13:23:55.4011352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size
2025-12-04T13:23:55.4011754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager
2025-12-04T13:23:55.4012135Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group
2025-12-04T13:23:55.4012510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum
2025-12-04T13:23:55.4012865Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice
2025-12-04T13:23:55.4013211Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks
2025-12-04T13:23:55.4013553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex
2025-12-04T13:23:55.4013896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2025-12-04T13:23:55.4014235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list
2025-12-04T13:23:55.4014624Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2025-12-04T13:23:55.4014998Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2025-12-04T13:23:55.4015369Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag
2025-12-04T13:23:55.4015741Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler
2025-12-04T13:23:55.4016171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-12-04T13:23:55.4016388Z 
2025-12-04T13:23:55.4016475Z Running distributed tests for the gloo backend with file init_method
2025-12-04T13:23:55.4016649Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:23:55.4017120Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:23:55.399006]
2025-12-04T13:27:12.4356248Z 
2025-12-04T13:27:12.4356969Z distributed/test_distributed_spawn 2/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.7_84d9f2e957f90adb_.log
2025-12-04T13:27:12.4363609Z Running 41 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-12-04T13:27:12.4370063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient
2025-12-04T13:27:12.4370538Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine
2025-12-04T13:27:12.4370983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2025-12-04T13:27:12.4371406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad
2025-12-04T13:27:12.4371810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2025-12-04T13:27:12.4372207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max
2025-12-04T13:27:12.4372611Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum
2025-12-04T13:27:12.4373009Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max
2025-12-04T13:27:12.4373391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max
2025-12-04T13:27:12.4373779Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max
2025-12-04T13:27:12.4374149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product
2025-12-04T13:27:12.4374514Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max
2025-12-04T13:27:12.4374860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min
2025-12-04T13:27:12.4375202Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum
2025-12-04T13:27:12.4375607Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async
2025-12-04T13:27:12.4376004Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2025-12-04T13:27:12.4376426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2025-12-04T13:27:12.4376854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters
2025-12-04T13:27:12.4377230Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier
2025-12-04T13:27:12.4377579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2025-12-04T13:27:12.4377939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast
2025-12-04T13:27:12.4378311Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks
2025-12-04T13:27:12.4378702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks
2025-12-04T13:27:12.4379066Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2025-12-04T13:27:12.4379403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank
2025-12-04T13:27:12.4379751Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2025-12-04T13:27:12.4380090Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv
2025-12-04T13:27:12.4380472Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size
2025-12-04T13:27:12.4380878Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager
2025-12-04T13:27:12.4381267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group
2025-12-04T13:27:12.4381649Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum
2025-12-04T13:27:12.4382007Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice
2025-12-04T13:27:12.4382352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks
2025-12-04T13:27:12.4382704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex
2025-12-04T13:27:12.4383050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2025-12-04T13:27:12.4383393Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list
2025-12-04T13:27:12.4383748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2025-12-04T13:27:12.4384126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2025-12-04T13:27:12.4384506Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag
2025-12-04T13:27:12.4384880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler
2025-12-04T13:27:12.4385319Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-12-04T13:27:12.4385540Z 
2025-12-04T13:27:12.4385673Z Finished distributed/test_distributed_spawn 2/7 ... [2025-12-04 13:27:12.436524][2238656.893843411], took 12.51min
2025-12-04T13:27:12.4386140Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:27:12.4388606Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:27:12.4390897Z Running distributed/test_distributed_spawn 5/7 ... [2025-12-04 13:27:12.439018][2238656.896343502]
2025-12-04T13:27:12.4391605Z MPI not available -- MPI backend tests will be skipped
2025-12-04T13:27:12.4392677Z Running distributed tests for the test backend with env init_method
2025-12-04T13:27:12.4393392Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:27:12.4395039Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:27:12.439368]
2025-12-04T13:27:14.3732273Z 
2025-12-04T13:27:14.3732971Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_298343d0f4f2338d_.log
2025-12-04T13:27:14.3733402Z Running 0 items in this shard:
2025-12-04T13:27:14.3733483Z 
2025-12-04T13:27:14.3737224Z Running distributed tests for the test backend with file init_method
2025-12-04T13:27:14.3738131Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:27:14.3740241Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:27:14.373895]
2025-12-04T13:27:16.2993854Z 
2025-12-04T13:27:16.2994653Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_625333d15483af50_.log
2025-12-04T13:27:16.2995241Z Running 0 items in this shard:
2025-12-04T13:27:16.2995379Z 
2025-12-04T13:27:16.3000454Z Running distributed tests for the nccl backend with env init_method
2025-12-04T13:27:16.3000830Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:27:16.3003884Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:27:16.300209]
2025-12-04T13:31:12.9576889Z 
2025-12-04T13:31:12.9578541Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_4276317b43b7335a_.log
2025-12-04T13:31:12.9595361Z Running 49 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-12-04T13:31:12.9605960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
2025-12-04T13:31:12.9606540Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2025-12-04T13:31:12.9607133Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2025-12-04T13:31:12.9607549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather
2025-12-04T13:31:12.9607952Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda
2025-12-04T13:31:12.9608377Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg
2025-12-04T13:31:12.9608841Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product
2025-12-04T13:31:12.9609306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min
2025-12-04T13:31:12.9609764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product
2025-12-04T13:31:12.9610215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum
2025-12-04T13:31:12.9610668Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops
2025-12-04T13:31:12.9611114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex
2025-12-04T13:31:12.9611532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda
2025-12-04T13:31:12.9611939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group
2025-12-04T13:31:12.9612375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex
2025-12-04T13:31:12.9612814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda
2025-12-04T13:31:12.9613197Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global
2025-12-04T13:31:12.9613566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags
2025-12-04T13:31:12.9613954Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err
2025-12-04T13:31:12.9614346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl
2025-12-04T13:31:12.9614748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2025-12-04T13:31:12.9615139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2025-12-04T13:31:12.9615568Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group
2025-12-04T13:31:12.9615956Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg
2025-12-04T13:31:12.9616316Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu
2025-12-04T13:31:12.9616692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler
2025-12-04T13:31:12.9617162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace
2025-12-04T13:31:12.9617538Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged
2025-12-04T13:31:12.9617916Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad
2025-12-04T13:31:12.9618309Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params
2025-12-04T13:31:12.9618698Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable
2025-12-04T13:31:12.9619075Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars
2025-12-04T13:31:12.9619437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
2025-12-04T13:31:12.9619783Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
2025-12-04T13:31:12.9620132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
2025-12-04T13:31:12.9620528Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
2025-12-04T13:31:12.9620944Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module
2025-12-04T13:31:12.9621385Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view
2025-12-04T13:31:12.9621817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2025-12-04T13:31:12.9622164Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
2025-12-04T13:31:12.9622498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
2025-12-04T13:31:12.9622843Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
2025-12-04T13:31:12.9623192Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
2025-12-04T13:31:12.9623531Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv
2025-12-04T13:31:12.9623904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
2025-12-04T13:31:12.9624304Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler
2025-12-04T13:31:12.9624684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-12-04T13:31:12.9625054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-12-04T13:31:12.9625470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-12-04T13:31:12.9625693Z 
2025-12-04T13:31:12.9625787Z Running distributed tests for the nccl backend with file init_method
2025-12-04T13:31:12.9625961Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:31:12.9626397Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:31:12.958922]
2025-12-04T13:35:08.3748010Z 
2025-12-04T13:35:08.3748900Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_358c0b7420749e6c_.log
2025-12-04T13:35:08.3767131Z Running 49 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-12-04T13:35:08.3780844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
2025-12-04T13:35:08.3781493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2025-12-04T13:35:08.3782059Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2025-12-04T13:35:08.3782520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather
2025-12-04T13:35:08.3782953Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda
2025-12-04T13:35:08.3783417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg
2025-12-04T13:35:08.3783928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product
2025-12-04T13:35:08.3784441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min
2025-12-04T13:35:08.3784944Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product
2025-12-04T13:35:08.3785437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum
2025-12-04T13:35:08.3785997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops
2025-12-04T13:35:08.3786483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex
2025-12-04T13:35:08.3787034Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda
2025-12-04T13:35:08.3787485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group
2025-12-04T13:35:08.3787962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex
2025-12-04T13:35:08.3788478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda
2025-12-04T13:35:08.3789010Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global
2025-12-04T13:35:08.3789482Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags
2025-12-04T13:35:08.3789930Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err
2025-12-04T13:35:08.3790327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl
2025-12-04T13:35:08.3790738Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2025-12-04T13:35:08.3791141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2025-12-04T13:35:08.3791548Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group
2025-12-04T13:35:08.3791938Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg
2025-12-04T13:35:08.3792301Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu
2025-12-04T13:35:08.3792681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler
2025-12-04T13:35:08.3793072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace
2025-12-04T13:35:08.3793450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged
2025-12-04T13:35:08.3793831Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad
2025-12-04T13:35:08.3794225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params
2025-12-04T13:35:08.3794647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable
2025-12-04T13:35:08.3795037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars
2025-12-04T13:35:08.3795408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
2025-12-04T13:35:08.3795761Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
2025-12-04T13:35:08.3796120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
2025-12-04T13:35:08.3796527Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
2025-12-04T13:35:08.3796999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module
2025-12-04T13:35:08.3797484Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view
2025-12-04T13:35:08.3797924Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2025-12-04T13:35:08.3798285Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
2025-12-04T13:35:08.3798630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
2025-12-04T13:35:08.3799013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
2025-12-04T13:35:08.3799371Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
2025-12-04T13:35:08.3799726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv
2025-12-04T13:35:08.3800094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
2025-12-04T13:35:08.3800488Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler
2025-12-04T13:35:08.3800867Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-12-04T13:35:08.3801235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-12-04T13:35:08.3801619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-12-04T13:35:08.3801842Z 
2025-12-04T13:35:08.3801935Z Running distributed tests for the gloo backend with env init_method
2025-12-04T13:35:08.3802112Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:35:08.3802548Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:35:08.376103]
2025-12-04T13:39:01.0423322Z 
2025-12-04T13:39:01.0424687Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_96aa5ce60cbfc7c9_.log
2025-12-04T13:39:01.0439514Z Running 49 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-12-04T13:39:01.0449207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
2025-12-04T13:39:01.0449768Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2025-12-04T13:39:01.0450219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2025-12-04T13:39:01.0450584Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather
2025-12-04T13:39:01.0450934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda
2025-12-04T13:39:01.0451349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg
2025-12-04T13:39:01.0451758Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product
2025-12-04T13:39:01.0452171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min
2025-12-04T13:39:01.0452578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product
2025-12-04T13:39:01.0452975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum
2025-12-04T13:39:01.0453370Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops
2025-12-04T13:39:01.0453767Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex
2025-12-04T13:39:01.0454131Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda
2025-12-04T13:39:01.0454492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group
2025-12-04T13:39:01.0454881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex
2025-12-04T13:39:01.0455316Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda
2025-12-04T13:39:01.0455708Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global
2025-12-04T13:39:01.0456085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags
2025-12-04T13:39:01.0456486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err
2025-12-04T13:39:01.0456929Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl
2025-12-04T13:39:01.0457342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2025-12-04T13:39:01.0457734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2025-12-04T13:39:01.0458109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group
2025-12-04T13:39:01.0458489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg
2025-12-04T13:39:01.0458843Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu
2025-12-04T13:39:01.0459210Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler
2025-12-04T13:39:01.0459627Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace
2025-12-04T13:39:01.0459995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged
2025-12-04T13:39:01.0460366Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad
2025-12-04T13:39:01.0460755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params
2025-12-04T13:39:01.0461177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable
2025-12-04T13:39:01.0461547Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars
2025-12-04T13:39:01.0461902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
2025-12-04T13:39:01.0462239Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
2025-12-04T13:39:01.0462581Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
2025-12-04T13:39:01.0462966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
2025-12-04T13:39:01.0463370Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module
2025-12-04T13:39:01.0463809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view
2025-12-04T13:39:01.0464228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2025-12-04T13:39:01.0464564Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
2025-12-04T13:39:01.0464895Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
2025-12-04T13:39:01.0465235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
2025-12-04T13:39:01.0465581Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
2025-12-04T13:39:01.0465917Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv
2025-12-04T13:39:01.0466281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
2025-12-04T13:39:01.0466675Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler
2025-12-04T13:39:01.0467206Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-12-04T13:39:01.0467575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-12-04T13:39:01.0467962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-12-04T13:39:01.0468190Z 
2025-12-04T13:39:01.0468278Z Running distributed tests for the gloo backend with file init_method
2025-12-04T13:39:01.0468456Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:39:01.0468927Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=7', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:39:01.042977]
2025-12-04T13:42:55.1026043Z 
2025-12-04T13:42:55.1026688Z distributed/test_distributed_spawn 5/7 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.7_408eed11c01653c5_.log
2025-12-04T13:42:55.1034775Z Running 49 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-12-04T13:42:55.1042535Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
2025-12-04T13:42:55.1043038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2025-12-04T13:42:55.1043477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2025-12-04T13:42:55.1043838Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather
2025-12-04T13:42:55.1044176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda
2025-12-04T13:42:55.1044543Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg
2025-12-04T13:42:55.1044939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product
2025-12-04T13:42:55.1045343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min
2025-12-04T13:42:55.1045733Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product
2025-12-04T13:42:55.1046121Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum
2025-12-04T13:42:55.1046501Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops
2025-12-04T13:42:55.1046921Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex
2025-12-04T13:42:55.1047275Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda
2025-12-04T13:42:55.1047626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group
2025-12-04T13:42:55.1047998Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex
2025-12-04T13:42:55.1048453Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda
2025-12-04T13:42:55.1048838Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global
2025-12-04T13:42:55.1049200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags
2025-12-04T13:42:55.1049583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err
2025-12-04T13:42:55.1050006Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl
2025-12-04T13:42:55.1050405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2025-12-04T13:42:55.1050797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2025-12-04T13:42:55.1051193Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group
2025-12-04T13:42:55.1051582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg
2025-12-04T13:42:55.1051935Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu
2025-12-04T13:42:55.1052305Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler
2025-12-04T13:42:55.1052694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace
2025-12-04T13:42:55.1053070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged
2025-12-04T13:42:55.1053447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad
2025-12-04T13:42:55.1053839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params
2025-12-04T13:42:55.1054224Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable
2025-12-04T13:42:55.1054597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars
2025-12-04T13:42:55.1054959Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
2025-12-04T13:42:55.1055299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
2025-12-04T13:42:55.1055644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
2025-12-04T13:42:55.1056033Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
2025-12-04T13:42:55.1056439Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module
2025-12-04T13:42:55.1056924Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view
2025-12-04T13:42:55.1057347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2025-12-04T13:42:55.1057688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
2025-12-04T13:42:55.1058057Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
2025-12-04T13:42:55.1058396Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
2025-12-04T13:42:55.1058737Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
2025-12-04T13:42:55.1059070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv
2025-12-04T13:42:55.1059437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
2025-12-04T13:42:55.1059863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler
2025-12-04T13:42:55.1060238Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-12-04T13:42:55.1060604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-12-04T13:42:55.1060985Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-12-04T13:42:55.1061209Z 
2025-12-04T13:42:55.1061343Z Finished distributed/test_distributed_spawn 5/7 ... [2025-12-04 13:42:55.103033][2239599.560353886], took 15.71min
2025-12-04T13:42:55.1061788Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:42:55.1062191Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:42:55.1062413Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-12-04T13:42:55.1062599Z Uploading artifacts took 0.00 seconds
2025-12-04T13:42:55.1062797Z Running distributed/fsdp/test_fsdp_input 1/1 ... [2025-12-04 13:42:55.105384][2239599.562708489]
2025-12-04T13:42:55.1062993Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:42:55.1063400Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:42:55.105537]
2025-12-04T13:43:51.8924446Z 
2025-12-04T13:43:51.8925467Z PRINTING LOG FILE of distributed/fsdp/test_fsdp_input 1/1 (test/test-reports/distributed.fsdp.test_fsdp_input_1.1_f3ac7cca0f55f6fb_.log)
2025-12-04T13:43:51.8926684Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-11107af51341f514.xml
2025-12-04T13:43:51.8927577Z ============================= test session starts ==============================
2025-12-04T13:43:51.8928198Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:43:51.8928725Z cachedir: .pytest_cache
2025-12-04T13:43:51.8929240Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:43:51.8929796Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:43:51.8930062Z configfile: pytest.ini
2025-12-04T13:43:51.8930585Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:43:51.8931148Z collecting ... collected 2 items
2025-12-04T13:43:51.8931467Z stepcurrent: Cannot find last run test, not skipping
2025-12-04T13:43:51.8932320Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda, test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda
2025-12-04T13:43:51.8933005Z 
2025-12-04T13:43:51.8934092Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda I1204 13:42:56.804000 310950 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 311019
2025-12-04T13:43:51.8935584Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T13:43:51.8936485Z   _init_core_state(
2025-12-04T13:43:51.8939432Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:43:51.8941832Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:43:51.8942323Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:43:51.8942859Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:43:51.8943638Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.8944394Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:43:51.8945153Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.8945860Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:43:51.8946551Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.8947368Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.8948097Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.8948819Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.8949546Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.8950193Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:43:51.8950845Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.8951406Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:43:51.8952190Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.8952944Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.8953365Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.8954037Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.8954624Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.8955058Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.8955559Z [rank0]:E1204 13:43:02.038000 311019 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:43:51.8955856Z dist init r=0, world=1
2025-12-04T13:43:51.8956365Z [rank0]:[W1204 13:43:02.698089068 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:43:51.8956890Z FAILED [6.7097s] [ 50%]
2025-12-04T13:43:51.8956968Z 
2025-12-04T13:43:51.8957039Z =================================== FAILURES ===================================
2025-12-04T13:43:51.8957253Z ___________________ TestInputCUDA.test_input_type_dict_cuda ____________________
2025-12-04T13:43:51.8957453Z Traceback (most recent call last):
2025-12-04T13:43:51.8957753Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:43:51.8958050Z     self._join_processes(fn)
2025-12-04T13:43:51.8966265Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:43:51.8966542Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:43:51.8966865Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:43:51.8967133Z     raise RuntimeError(error)
2025-12-04T13:43:51.8967291Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.8967457Z Traceback (most recent call last):
2025-12-04T13:43:51.8967706Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.8967953Z     getattr(self, test_name)()
2025-12-04T13:43:51.8968193Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.8968429Z     fn()
2025-12-04T13:43:51.8968633Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.8968866Z     method(*args, **kwargs)
2025-12-04T13:43:51.8969163Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.8969396Z     method(*args, **kwargs)
2025-12-04T13:43:51.8969616Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.8969843Z     with policy():
2025-12-04T13:43:51.8970061Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.8970294Z     raise RuntimeError(msg)
2025-12-04T13:43:51.8970704Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.8971044Z 
2025-12-04T13:43:51.8971126Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.8971436Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.8971659Z 
2025-12-04T13:43:51.8971753Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.8971883Z 
2025-12-04T13:43:51.8971885Z 
2025-12-04T13:43:51.8971966Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:43:51.8972173Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:43:51.8972544Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-11107af51341f514.xml -
2025-12-04T13:43:51.8972882Z =========================== short test summary info ============================
2025-12-04T13:43:51.8973189Z FAILED [6.7097s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.8973482Z Traceback (most recent call last):
2025-12-04T13:43:51.8973731Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.8973978Z     getattr(self, test_name)()
2025-12-04T13:43:51.8974213Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.8974447Z     fn()
2025-12-04T13:43:51.8974653Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.8974892Z     method(*args, **kwargs)
2025-12-04T13:43:51.8975120Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.8975356Z     method(*args, **kwargs)
2025-12-04T13:43:51.8975587Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.8975824Z     with policy():
2025-12-04T13:43:51.8976046Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.8976286Z     raise RuntimeError(msg)
2025-12-04T13:43:51.8976663Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.8977062Z 
2025-12-04T13:43:51.8977139Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.8977447Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.8977679Z 
2025-12-04T13:43:51.8977770Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.8978013Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:43:51.8978183Z ============================== 1 failed in 6.72s ===============================
2025-12-04T13:43:51.8978330Z Got exit code 1
2025-12-04T13:43:51.8978437Z Retrying single test...
2025-12-04T13:43:51.8978707Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-3ad14c89cda265f9.xml
2025-12-04T13:43:51.8979003Z ============================= test session starts ==============================
2025-12-04T13:43:51.8979258Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:43:51.8979456Z cachedir: .pytest_cache
2025-12-04T13:43:51.8979689Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:43:51.8979938Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:43:51.8980070Z configfile: pytest.ini
2025-12-04T13:43:51.8980316Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:43:51.8980597Z collecting ... collected 2 items / 1 deselected / 1 selected
2025-12-04T13:43:51.8980896Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda
2025-12-04T13:43:51.8981163Z Running 1 items in this shard
2025-12-04T13:43:51.8981249Z 
2025-12-04T13:43:51.8981520Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda I1204 13:43:05.760000 311102 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 311171
2025-12-04T13:43:51.8982142Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T13:43:51.8982526Z   _init_core_state(
2025-12-04T13:43:51.8983889Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:43:51.8985328Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:43:51.8985639Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:43:51.8985985Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:43:51.8986485Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.8987023Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:43:51.8987546Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.8987999Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:43:51.8988444Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.8988954Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.8989428Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.8989900Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.8990369Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.8990830Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:43:51.8991299Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.8991769Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:43:51.8992397Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.8992985Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.8993350Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.8993919Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.8994393Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.8994769Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.8995193Z [rank0]:E1204 13:43:10.939000 311171 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:43:51.8995440Z dist init r=0, world=1
2025-12-04T13:43:51.8995853Z [rank0]:[W1204 13:43:11.692949841 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:43:51.8996271Z FAILED [6.8089s] [100%]
2025-12-04T13:43:51.8996336Z 
2025-12-04T13:43:51.8996402Z =================================== FAILURES ===================================
2025-12-04T13:43:51.8996593Z ___________________ TestInputCUDA.test_input_type_dict_cuda ____________________
2025-12-04T13:43:51.8996837Z Traceback (most recent call last):
2025-12-04T13:43:51.8997094Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:43:51.8997351Z     self._join_processes(fn)
2025-12-04T13:43:51.8997606Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:43:51.8997880Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:43:51.8998187Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:43:51.8998459Z     raise RuntimeError(error)
2025-12-04T13:43:51.8998620Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.8998792Z Traceback (most recent call last):
2025-12-04T13:43:51.8999046Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.8999299Z     getattr(self, test_name)()
2025-12-04T13:43:51.8999543Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.8999779Z     fn()
2025-12-04T13:43:51.8999984Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9000217Z     method(*args, **kwargs)
2025-12-04T13:43:51.9000442Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9000675Z     method(*args, **kwargs)
2025-12-04T13:43:51.9000896Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9001124Z     with policy():
2025-12-04T13:43:51.9001342Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9001576Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9001950Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9002288Z 
2025-12-04T13:43:51.9002368Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9002668Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.9002893Z 
2025-12-04T13:43:51.9002980Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9003108Z 
2025-12-04T13:43:51.9003110Z 
2025-12-04T13:43:51.9003187Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:43:51.9003392Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:43:51.9003754Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-3ad14c89cda265f9.xml -
2025-12-04T13:43:51.9004091Z =========================== short test summary info ============================
2025-12-04T13:43:51.9004394Z FAILED [6.8089s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9004683Z Traceback (most recent call last):
2025-12-04T13:43:51.9004933Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9005179Z     getattr(self, test_name)()
2025-12-04T13:43:51.9005443Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9005679Z     fn()
2025-12-04T13:43:51.9005881Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9006112Z     method(*args, **kwargs)
2025-12-04T13:43:51.9006333Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9006564Z     method(*args, **kwargs)
2025-12-04T13:43:51.9006819Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9007080Z     with policy():
2025-12-04T13:43:51.9007293Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9007526Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9007902Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9008244Z 
2025-12-04T13:43:51.9008319Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9008618Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.9008843Z 
2025-12-04T13:43:51.9008934Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9009127Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:43:51.9009294Z ======================= 1 failed, 1 deselected in 6.82s ========================
2025-12-04T13:43:51.9009431Z Got exit code 1
2025-12-04T13:43:51.9009532Z Retrying single test...
2025-12-04T13:43:51.9009797Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-a23d2fe3cf27e077.xml
2025-12-04T13:43:51.9010086Z ============================= test session starts ==============================
2025-12-04T13:43:51.9010301Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:43:51.9010494Z cachedir: .pytest_cache
2025-12-04T13:43:51.9010719Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:43:51.9010964Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:43:51.9011087Z configfile: pytest.ini
2025-12-04T13:43:51.9011314Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:43:51.9011585Z collecting ... collected 2 items / 1 deselected / 1 selected
2025-12-04T13:43:51.9011876Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda
2025-12-04T13:43:51.9012133Z Running 1 items in this shard
2025-12-04T13:43:51.9012208Z 
2025-12-04T13:43:51.9012476Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda I1204 13:43:14.807000 311254 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 311323
2025-12-04T13:43:51.9013080Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T13:43:51.9013455Z   _init_core_state(
2025-12-04T13:43:51.9014828Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:43:51.9016281Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:43:51.9016590Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:43:51.9016979Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:43:51.9017478Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9017965Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:43:51.9018454Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9018909Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:43:51.9019358Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9019833Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.9020304Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9020773Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.9021242Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9021702Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:43:51.9022162Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9022636Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:43:51.9023267Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9023854Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.9024249Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9024804Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.9025271Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.9025681Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9026106Z [rank0]:E1204 13:43:20.009000 311323 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:43:51.9026358Z dist init r=0, world=1
2025-12-04T13:43:51.9026797Z [rank0]:[W1204 13:43:20.689846490 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:43:51.9027215Z FAILED [6.8089s] [100%]
2025-12-04T13:43:51.9027280Z 
2025-12-04T13:43:51.9027342Z =================================== FAILURES ===================================
2025-12-04T13:43:51.9027531Z ___________________ TestInputCUDA.test_input_type_dict_cuda ____________________
2025-12-04T13:43:51.9027702Z Traceback (most recent call last):
2025-12-04T13:43:51.9027953Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:43:51.9028201Z     self._join_processes(fn)
2025-12-04T13:43:51.9028452Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:43:51.9028720Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:43:51.9028990Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:43:51.9029254Z     raise RuntimeError(error)
2025-12-04T13:43:51.9029408Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9029573Z Traceback (most recent call last):
2025-12-04T13:43:51.9029818Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9030064Z     getattr(self, test_name)()
2025-12-04T13:43:51.9030301Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9030539Z     fn()
2025-12-04T13:43:51.9030748Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9030984Z     method(*args, **kwargs)
2025-12-04T13:43:51.9031211Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9031447Z     method(*args, **kwargs)
2025-12-04T13:43:51.9031672Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9031904Z     with policy():
2025-12-04T13:43:51.9032122Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9032359Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9032736Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9033106Z 
2025-12-04T13:43:51.9033187Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9033490Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.9033715Z 
2025-12-04T13:43:51.9033809Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9033935Z 
2025-12-04T13:43:51.9033936Z 
2025-12-04T13:43:51.9034048Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:43:51.9034253Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:43:51.9034621Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-a23d2fe3cf27e077.xml -
2025-12-04T13:43:51.9034962Z =========================== short test summary info ============================
2025-12-04T13:43:51.9035274Z FAILED [6.8089s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9035565Z Traceback (most recent call last):
2025-12-04T13:43:51.9035814Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9036063Z     getattr(self, test_name)()
2025-12-04T13:43:51.9036301Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9036544Z     fn()
2025-12-04T13:43:51.9036787Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9037022Z     method(*args, **kwargs)
2025-12-04T13:43:51.9037247Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9037483Z     method(*args, **kwargs)
2025-12-04T13:43:51.9037706Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9037937Z     with policy():
2025-12-04T13:43:51.9038152Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9038387Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9038763Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9039107Z 
2025-12-04T13:43:51.9039184Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9039486Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda
2025-12-04T13:43:51.9039714Z 
2025-12-04T13:43:51.9039803Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9039994Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:43:51.9040164Z ======================= 1 failed, 1 deselected in 6.82s ========================
2025-12-04T13:43:51.9040306Z Got exit code 1
2025-12-04T13:43:51.9040501Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda
2025-12-04T13:43:51.9040806Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T13:43:51.9041164Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-89b9ce9896b95eea.xml
2025-12-04T13:43:51.9041454Z ============================= test session starts ==============================
2025-12-04T13:43:51.9041701Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:43:51.9041896Z cachedir: .pytest_cache
2025-12-04T13:43:51.9042126Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:43:51.9042369Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:43:51.9042494Z configfile: pytest.ini
2025-12-04T13:43:51.9042725Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:43:51.9043033Z collecting ... collected 2 items / 1 deselected / 1 selected
2025-12-04T13:43:51.9043197Z stepcurrent: skipping 1 already run items.
2025-12-04T13:43:51.9043332Z Running 1 items in this shard
2025-12-04T13:43:51.9043409Z 
2025-12-04T13:43:51.9043686Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda I1204 13:43:23.877000 311406 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 311475
2025-12-04T13:43:51.9044292Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T13:43:51.9044671Z   _init_core_state(
2025-12-04T13:43:51.9046015Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:43:51.9047479Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:43:51.9047790Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:43:51.9048139Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:43:51.9048638Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9049127Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:43:51.9049614Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9050076Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:43:51.9050526Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9050995Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.9051497Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9051967Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.9052440Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9052927Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:43:51.9053393Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9053865Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:43:51.9054495Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9055082Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.9055439Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9055992Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9056463Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.9056868Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9057292Z [rank0]:E1204 13:43:29.088000 311475 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:43:51.9057540Z dist init r=0, world=1
2025-12-04T13:43:51.9057946Z [rank0]:[W1204 13:43:29.867675768 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:43:51.9058360Z FAILED [6.9097s] [100%]
2025-12-04T13:43:51.9058423Z 
2025-12-04T13:43:51.9058484Z =================================== FAILURES ===================================
2025-12-04T13:43:51.9058665Z ___________________ TestInputCUDA.test_input_type_list_cuda ____________________
2025-12-04T13:43:51.9058834Z Traceback (most recent call last):
2025-12-04T13:43:51.9059084Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:43:51.9059335Z     self._join_processes(fn)
2025-12-04T13:43:51.9059588Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:43:51.9059858Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:43:51.9060131Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:43:51.9060437Z     raise RuntimeError(error)
2025-12-04T13:43:51.9060594Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9060760Z Traceback (most recent call last):
2025-12-04T13:43:51.9061005Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9061253Z     getattr(self, test_name)()
2025-12-04T13:43:51.9061490Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9061758Z     fn()
2025-12-04T13:43:51.9061965Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9062202Z     method(*args, **kwargs)
2025-12-04T13:43:51.9062429Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9062671Z     method(*args, **kwargs)
2025-12-04T13:43:51.9062897Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9063130Z     with policy():
2025-12-04T13:43:51.9063348Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9063587Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9063966Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9064309Z 
2025-12-04T13:43:51.9064391Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9064695Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9064924Z 
2025-12-04T13:43:51.9065014Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9065144Z 
2025-12-04T13:43:51.9065146Z 
2025-12-04T13:43:51.9065224Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:43:51.9065429Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:43:51.9065796Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-89b9ce9896b95eea.xml -
2025-12-04T13:43:51.9066139Z =========================== short test summary info ============================
2025-12-04T13:43:51.9066446Z FAILED [6.9097s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9066738Z Traceback (most recent call last):
2025-12-04T13:43:51.9067037Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9067286Z     getattr(self, test_name)()
2025-12-04T13:43:51.9067525Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9067760Z     fn()
2025-12-04T13:43:51.9067963Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9068194Z     method(*args, **kwargs)
2025-12-04T13:43:51.9068413Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9068641Z     method(*args, **kwargs)
2025-12-04T13:43:51.9068857Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9069082Z     with policy():
2025-12-04T13:43:51.9069369Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9069602Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9069982Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9070326Z 
2025-12-04T13:43:51.9070402Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9070733Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9070957Z 
2025-12-04T13:43:51.9071046Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9071238Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:43:51.9071412Z ======================= 1 failed, 1 deselected in 6.92s ========================
2025-12-04T13:43:51.9071557Z Got exit code 1
2025-12-04T13:43:51.9071658Z Retrying single test...
2025-12-04T13:43:51.9071919Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-eccc1a84130befcf.xml
2025-12-04T13:43:51.9072211Z ============================= test session starts ==============================
2025-12-04T13:43:51.9072425Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:43:51.9072620Z cachedir: .pytest_cache
2025-12-04T13:43:51.9072851Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:43:51.9073090Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:43:51.9073213Z configfile: pytest.ini
2025-12-04T13:43:51.9073444Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:43:51.9073711Z collecting ... collected 2 items / 1 deselected / 1 selected
2025-12-04T13:43:51.9073994Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda
2025-12-04T13:43:51.9074251Z Running 1 items in this shard
2025-12-04T13:43:51.9074329Z 
2025-12-04T13:43:51.9074598Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda I1204 13:43:33.088000 311558 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 311627
2025-12-04T13:43:51.9075205Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T13:43:51.9075599Z   _init_core_state(
2025-12-04T13:43:51.9076981Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:43:51.9078442Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:43:51.9078748Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:43:51.9079092Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:43:51.9079589Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9080110Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:43:51.9080596Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9081045Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:43:51.9081489Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9081953Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.9082423Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9082889Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.9083359Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9083816Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:43:51.9084276Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9084747Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:43:51.9085377Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9085956Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.9086311Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9086892Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9087357Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.9087764Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9088184Z [rank0]:E1204 13:43:38.289000 311627 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:43:51.9088432Z dist init r=0, world=1
2025-12-04T13:43:51.9088838Z [rank0]:[W1204 13:43:38.941770298 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:43:51.9089294Z FAILED [6.7099s] [100%]
2025-12-04T13:43:51.9089364Z 
2025-12-04T13:43:51.9089423Z =================================== FAILURES ===================================
2025-12-04T13:43:51.9089605Z ___________________ TestInputCUDA.test_input_type_list_cuda ____________________
2025-12-04T13:43:51.9089779Z Traceback (most recent call last):
2025-12-04T13:43:51.9090032Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:43:51.9090281Z     self._join_processes(fn)
2025-12-04T13:43:51.9090532Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:43:51.9090804Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:43:51.9091076Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:43:51.9091343Z     raise RuntimeError(error)
2025-12-04T13:43:51.9091500Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9091665Z Traceback (most recent call last):
2025-12-04T13:43:51.9091907Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9092156Z     getattr(self, test_name)()
2025-12-04T13:43:51.9092393Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9092630Z     fn()
2025-12-04T13:43:51.9092832Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9093061Z     method(*args, **kwargs)
2025-12-04T13:43:51.9093286Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9093522Z     method(*args, **kwargs)
2025-12-04T13:43:51.9093746Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9093978Z     with policy():
2025-12-04T13:43:51.9094195Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9094434Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9094806Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9095144Z 
2025-12-04T13:43:51.9095218Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9095519Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9095748Z 
2025-12-04T13:43:51.9095838Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9095967Z 
2025-12-04T13:43:51.9095968Z 
2025-12-04T13:43:51.9096046Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:43:51.9096249Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:43:51.9096650Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-eccc1a84130befcf.xml -
2025-12-04T13:43:51.9097028Z =========================== short test summary info ============================
2025-12-04T13:43:51.9097330Z FAILED [6.7099s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9097614Z Traceback (most recent call last):
2025-12-04T13:43:51.9097894Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9098138Z     getattr(self, test_name)()
2025-12-04T13:43:51.9098370Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9098600Z     fn()
2025-12-04T13:43:51.9098807Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9099042Z     method(*args, **kwargs)
2025-12-04T13:43:51.9099260Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9099488Z     method(*args, **kwargs)
2025-12-04T13:43:51.9099705Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9099932Z     with policy():
2025-12-04T13:43:51.9100142Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9100372Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9100744Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9101080Z 
2025-12-04T13:43:51.9101156Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9101451Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9101672Z 
2025-12-04T13:43:51.9101762Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9101951Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:43:51.9102117Z ======================= 1 failed, 1 deselected in 6.72s ========================
2025-12-04T13:43:51.9102254Z Got exit code 1
2025-12-04T13:43:51.9102356Z Retrying single test...
2025-12-04T13:43:51.9102617Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-1e00d05fd4b3b7d9.xml
2025-12-04T13:43:51.9102909Z ============================= test session starts ==============================
2025-12-04T13:43:51.9103120Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:43:51.9103315Z cachedir: .pytest_cache
2025-12-04T13:43:51.9103543Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:43:51.9103780Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:43:51.9103899Z configfile: pytest.ini
2025-12-04T13:43:51.9104128Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:43:51.9104395Z collecting ... collected 2 items / 1 deselected / 1 selected
2025-12-04T13:43:51.9104680Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda
2025-12-04T13:43:51.9104933Z Running 1 items in this shard
2025-12-04T13:43:51.9105005Z 
2025-12-04T13:43:51.9105307Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda I1204 13:43:42.069000 311710 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 311779
2025-12-04T13:43:51.9105906Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T13:43:51.9106300Z   _init_core_state(
2025-12-04T13:43:51.9107668Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:43:51.9109098Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:43:51.9109402Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:43:51.9109743Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:43:51.9110235Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9110717Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:43:51.9111196Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9111646Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:43:51.9112097Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9112563Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.9113026Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9113487Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:43:51.9113952Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9114408Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:43:51.9114898Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9115367Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:43:51.9115995Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9116605Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.9116995Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9117547Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9118008Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:43:51.9118374Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9118789Z [rank0]:E1204 13:43:47.307000 311779 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:43:51.9119032Z dist init r=0, world=1
2025-12-04T13:43:51.9119434Z [rank0]:[W1204 13:43:47.058892752 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:43:51.9119844Z FAILED [6.9099s] [100%]
2025-12-04T13:43:51.9119907Z 
2025-12-04T13:43:51.9119963Z =================================== FAILURES ===================================
2025-12-04T13:43:51.9120140Z ___________________ TestInputCUDA.test_input_type_list_cuda ____________________
2025-12-04T13:43:51.9120307Z Traceback (most recent call last):
2025-12-04T13:43:51.9120550Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:43:51.9120793Z     self._join_processes(fn)
2025-12-04T13:43:51.9121036Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:43:51.9121299Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:43:51.9121565Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:43:51.9121822Z     raise RuntimeError(error)
2025-12-04T13:43:51.9121970Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9122129Z Traceback (most recent call last):
2025-12-04T13:43:51.9122368Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9122612Z     getattr(self, test_name)()
2025-12-04T13:43:51.9122841Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9123070Z     fn()
2025-12-04T13:43:51.9123269Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9123495Z     method(*args, **kwargs)
2025-12-04T13:43:51.9123748Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9123976Z     method(*args, **kwargs)
2025-12-04T13:43:51.9124192Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9124413Z     with policy():
2025-12-04T13:43:51.9124622Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9124892Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9125262Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9125599Z 
2025-12-04T13:43:51.9125677Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9125972Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9126196Z 
2025-12-04T13:43:51.9126284Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9126416Z 
2025-12-04T13:43:51.9126417Z 
2025-12-04T13:43:51.9126498Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:43:51.9126706Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:43:51.9127113Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-1e00d05fd4b3b7d9.xml -
2025-12-04T13:43:51.9127453Z =========================== short test summary info ============================
2025-12-04T13:43:51.9127766Z FAILED [6.9099s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:43:51.9128057Z Traceback (most recent call last):
2025-12-04T13:43:51.9128310Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:43:51.9128559Z     getattr(self, test_name)()
2025-12-04T13:43:51.9128796Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:43:51.9129037Z     fn()
2025-12-04T13:43:51.9129242Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9129477Z     method(*args, **kwargs)
2025-12-04T13:43:51.9129702Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:43:51.9129937Z     method(*args, **kwargs)
2025-12-04T13:43:51.9130162Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:43:51.9130393Z     with policy():
2025-12-04T13:43:51.9130609Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:43:51.9130846Z     raise RuntimeError(msg)
2025-12-04T13:43:51.9131222Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 1633681408 and is now 2130706432.
2025-12-04T13:43:51.9131563Z 
2025-12-04T13:43:51.9131643Z To execute this test, run the following from the base repo dir:
2025-12-04T13:43:51.9131946Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda
2025-12-04T13:43:51.9132168Z 
2025-12-04T13:43:51.9132298Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:43:51.9132493Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:43:51.9132662Z ======================= 1 failed, 1 deselected in 6.92s ========================
2025-12-04T13:43:51.9132802Z Got exit code 1
2025-12-04T13:43:51.9132999Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda
2025-12-04T13:43:51.9133302Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T13:43:51.9133695Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-cfabbd43b3605d29.xml
2025-12-04T13:43:51.9133989Z ============================= test session starts ==============================
2025-12-04T13:43:51.9134199Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:43:51.9134397Z cachedir: .pytest_cache
2025-12-04T13:43:51.9134624Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:43:51.9134865Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:43:51.9134986Z configfile: pytest.ini
2025-12-04T13:43:51.9135215Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:43:51.9135486Z collecting ... collected 2 items / 2 deselected / 0 selected
2025-12-04T13:43:51.9135653Z stepcurrent: skipping 2 already run items.
2025-12-04T13:43:51.9135787Z Running 0 items in this shard
2025-12-04T13:43:51.9135860Z 
2025-12-04T13:43:51.9136102Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-cfabbd43b3605d29.xml -
2025-12-04T13:43:51.9136445Z ============================ 2 deselected in 0.00s =============================
2025-12-04T13:43:51.9136893Z The following tests failed consistently: ['test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda', 'test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda']
2025-12-04T13:43:51.9137211Z 
2025-12-04T13:43:51.9137404Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input 1/1 (test/test-reports/distributed.fsdp.test_fsdp_input_1.1_f3ac7cca0f55f6fb_.log)
2025-12-04T13:43:51.9137632Z 
2025-12-04T13:43:51.9137757Z Finished distributed/fsdp/test_fsdp_input 1/1 ... [2025-12-04 13:43:51.892541][2239656.349862533], took 0.95min
2025-12-04T13:43:51.9138200Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:43:51.9138598Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:43:51.9138823Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-12-04T13:43:51.9139008Z Uploading artifacts took 0.00 seconds
2025-12-04T13:43:51.9139149Z distributed/fsdp/test_fsdp_input 1/1 failed!
2025-12-04T13:43:51.9139375Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2025-12-04 13:43:51.895532][2239656.352856715]
2025-12-04T13:43:51.9139600Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:43:51.9140029Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:43:51.895740]
2025-12-04T13:47:00.1471063Z 
2025-12-04T13:47:00.1472262Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_9e7f904f370732b4_.log
2025-12-04T13:47:00.1482695Z Running 22 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp16, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_manual_reshard_with_reshard_after_forward_false, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reduce_scatter_divide_factor, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reshard_after_forward, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_backward_misprefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_unused_module, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch_inside_ac, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_forward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiProcess::test_unshard_async, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_no_param_group, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_without_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardAllocFromPG::test_exception_when_used_together_with_comm_hooks, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardAllocFromPG::test_fully_shard_alloc_from_pg, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardForceSumReduction::test_fully_shard_force_sum_both_reductions, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardForceSumReduction::test_fully_shard_force_sum_reduce_scatter, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardReduceOpWorldSize1::test_size1_reduceop
2025-12-04T13:47:00.1493265Z 
2025-12-04T13:47:00.1493548Z Finished distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2025-12-04 13:47:00.149052][2239844.606372934], took 3.14min
2025-12-04T13:47:00.1507807Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:47:00.1514360Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:47:00.1517634Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2025-12-04 13:47:00.151645][2239844.608969823]
2025-12-04T13:47:00.1517903Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:47:00.1519466Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:47:00.151823]
2025-12-04T13:47:35.4156490Z 
2025-12-04T13:47:35.4158171Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_2412ca35482dbd45_.log
2025-12-04T13:47:35.4162169Z Running 9 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions0, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions1, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions2, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithCaching::test_read_write_shard_tensor
2025-12-04T13:47:35.4165896Z 
2025-12-04T13:47:35.4166186Z Finished distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2025-12-04 13:47:35.415406][2239879.872724776], took 0.59min
2025-12-04T13:47:35.4177295Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:47:35.4186394Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:47:35.4188275Z Running distributed/test_composability 1/1 ... [2025-12-04 13:47:35.418663][2239879.875985864]
2025-12-04T13:47:35.4188888Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:47:35.4190230Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:47:35.418858]
2025-12-04T13:47:48.3006666Z 
2025-12-04T13:47:48.3007861Z distributed/test_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_composability_1.1_80f51130a1074bf1_.log
2025-12-04T13:47:48.3013759Z Running 13 items in this shard: test/distributed/test_composability.py::ComposabilityTest::test_pp_ddp_ScheduleClass0, test/distributed/test_composability.py::ComposabilityTest::test_pp_ddp_ScheduleClass1, test/distributed/test_composability.py::ComposabilityTest::test_pp_ddp_ScheduleClass2, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_MP_ScheduleClass0, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_MP_ScheduleClass1, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_MP_ScheduleClass2, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_MP_ScheduleClass3, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_ScheduleClass0, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_ScheduleClass1, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_ScheduleClass2, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_ScheduleClass3, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_unshard_reshard_runtime_dp_type_FSDP, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_unshard_reshard_runtime_dp_type_FSDP_MP
2025-12-04T13:47:48.3015752Z 
2025-12-04T13:47:48.3015886Z Finished distributed/test_composability 1/1 ... [2025-12-04 13:47:48.300285][2239892.757605719], took 0.21min
2025-12-04T13:47:48.3020626Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:47:48.3027311Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:47:48.3030471Z Running distributed/test_multi_threaded_pg 1/1 ... [2025-12-04 13:47:48.302955][2239892.760278637]
2025-12-04T13:47:48.3030681Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:47:48.3032497Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:47:48.303164]
2025-12-04T13:47:50.8207327Z 
2025-12-04T13:47:50.8208522Z distributed/test_multi_threaded_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_multi_threaded_pg_1.1_9dc3b759b8f515bc_.log
2025-12-04T13:47:50.8216054Z Running 22 items in this shard: test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_none, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_tensor, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_skip, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_coalesced, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_ops, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_to_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_allgather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_assert_equal_on_rank, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_bwd_sees_fwd_pg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_gather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_reduce_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_subpg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_using_pg_from_another_thread
2025-12-04T13:47:50.8220887Z 
2025-12-04T13:47:50.8221091Z Finished distributed/test_multi_threaded_pg 1/1 ... [2025-12-04 13:47:50.820359][2239895.27767917], took 0.04min
2025-12-04T13:47:50.8221792Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:47:50.8228436Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:47:50.8231505Z Running distributed/_composable/fsdp/test_fully_shard_extensions 1/1 ... [2025-12-04 13:47:50.823055][2239895.280378407]
2025-12-04T13:47:50.8231791Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:47:50.8234316Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_extensions.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:47:50.823262]
2025-12-04T13:48:15.7755985Z 
2025-12-04T13:48:15.7756898Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_extensions_1.1_7ff0740c880751fd_.log
2025-12-04T13:48:15.7759528Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiProcess::test_all_gather_extensions_train_parity, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_hsdp_mesh, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_outer_size_stride, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_end_to_end, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_monkey_patch
2025-12-04T13:48:15.7761025Z 
2025-12-04T13:48:15.7761235Z Finished distributed/_composable/fsdp/test_fully_shard_extensions 1/1 ... [2025-12-04 13:48:15.775298][2239920.232614953], took 0.42min
2025-12-04T13:48:15.7777818Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:48:15.7785296Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:48:15.7790731Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2025-12-04 13:48:15.778752][2239920.236075798]
2025-12-04T13:48:15.7790993Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:48:15.7791450Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:48:15.778943]
2025-12-04T13:48:39.7310821Z 
2025-12-04T13:48:39.7312151Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_fb8c0eab87cffc7a_.log
2025-12-04T13:48:39.7323263Z Running 16 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadRot13::test_read_write_tensor_and_blob_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadRot13::test_read_write_tensor_and_blob_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadZStandard::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadZStandard::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2
2025-12-04T13:48:39.7328404Z 
2025-12-04T13:48:39.7328569Z Finished distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2025-12-04 13:48:39.730900][2239944.188217527], took 0.40min
2025-12-04T13:48:39.7330186Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:48:39.7337995Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:48:39.7340594Z Running distributed/fsdp/test_wrap 1/1 ... [2025-12-04 13:48:39.733956][2239944.191280878]
2025-12-04T13:48:39.7341138Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:48:39.7343757Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:48:39.734150]
2025-12-04T13:50:54.9099662Z 
2025-12-04T13:50:54.9100849Z distributed/fsdp/test_wrap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_wrap_1.1_f181d4302d8b55c8_.log
2025-12-04T13:50:54.9118352Z Running 52 items in this shard: test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_bn_always_wrapped_individually, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_False, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_True, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_zero_argument, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_api, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap_include_children, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf_custom, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_custom_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_frozen_params, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy_callable, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_transformer_auto_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_disabled_outside_context, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_override_defaults, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method1, test/distributed/fsdp/test_wrap.py::TestWrapUtils::test_validate_frozen_params
2025-12-04T13:50:54.9128908Z 
2025-12-04T13:50:54.9129100Z Finished distributed/fsdp/test_wrap 1/1 ... [2025-12-04 13:50:54.909733][2240079.367053719], took 2.25min
2025-12-04T13:50:54.9129638Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:50:54.9130113Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:50:54.9130422Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2025-12-04 13:50:54.912651][2240079.369975932]
2025-12-04T13:50:54.9130720Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:50:54.9131220Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:50:54.912829]
2025-12-04T13:53:25.8758037Z 
2025-12-04T13:53:25.8758859Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_71c045d8501db5d3_.log
2025-12-04T13:53:25.8764377Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf
2025-12-04T13:53:25.8769183Z 
2025-12-04T13:53:25.8769355Z Finished distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2025-12-04 13:53:25.875579][2240230.332899825], took 2.52min
2025-12-04T13:53:25.8776361Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:53:25.8782401Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:53:25.8785061Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2025-12-04 13:53:25.878408][2240230.335733]
2025-12-04T13:53:25.8785284Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:53:25.8786946Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:53:25.878582]
2025-12-04T13:53:58.8005270Z 
2025-12-04T13:53:58.8006508Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_5b4e29db20d16944_.log
2025-12-04T13:53:58.8008925Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state
2025-12-04T13:53:58.8010323Z 
2025-12-04T13:53:58.8010750Z Finished distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2025-12-04 13:53:58.800204][2240263.257525152], took 0.55min
2025-12-04T13:53:58.8025950Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:53:58.8032791Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:53:58.8035277Z Running distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2025-12-04 13:53:58.803442][2240263.26076712]
2025-12-04T13:53:58.8035622Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:53:58.8037617Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:53:58.803619]
2025-12-04T13:54:09.8864981Z 
2025-12-04T13:54:09.8866280Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_7c1541a10ab17e5e_.log
2025-12-04T13:54:09.8868369Z Running 2 items in this shard: test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_named_params_with_sharded_tensor, test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_sharded_optim
2025-12-04T13:54:09.8869462Z 
2025-12-04T13:54:09.8869905Z Finished distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2025-12-04 13:54:09.886254][2240274.34357412], took 0.18min
2025-12-04T13:54:09.8887192Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:54:09.8892727Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:54:09.8894914Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2025-12-04 13:54:09.889349][2240274.346673711]
2025-12-04T13:54:09.8895197Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:54:09.8896498Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:54:09.889517]
2025-12-04T13:54:50.2176212Z 
2025-12-04T13:54:50.2177736Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_3d4a5e4103869ef4_.log
2025-12-04T13:54:50.2180095Z Running 7 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_2d_state_dict_correctness, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_cached_state_dict, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_cpu_offload, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_hsdp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiThread::test_rank0_offload_full_state_dict
2025-12-04T13:54:50.2181527Z 
2025-12-04T13:54:50.2181704Z Finished distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2025-12-04 13:54:50.217301][2240314.674621193], took 0.67min
2025-12-04T13:54:50.2197065Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:54:50.2203761Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:54:50.2206189Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2025-12-04 13:54:50.220538][2240314.677862722]
2025-12-04T13:54:50.2206418Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:54:50.2208318Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:54:50.220719]
2025-12-04T13:57:16.2520205Z 
2025-12-04T13:57:16.2521317Z PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm 1/1 (test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_5e6bd88c9575523a_.log)
2025-12-04T13:57:16.2522649Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-1c423d878373b38f.xml
2025-12-04T13:57:16.2523533Z ============================= test session starts ==============================
2025-12-04T13:57:16.2524153Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.2524703Z cachedir: .pytest_cache
2025-12-04T13:57:16.2525337Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.2526005Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.2526346Z configfile: pytest.ini
2025-12-04T13:57:16.2527077Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.2528463Z collecting ... collected 4 items
2025-12-04T13:57:16.2528866Z stepcurrent: Cannot find last run test, not skipping
2025-12-04T13:57:16.2530718Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda
2025-12-04T13:57:16.2532085Z 
2025-12-04T13:57:16.2532547Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda I1204 13:54:51.944000 341159 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 341228
2025-12-04T13:57:16.2533320Z I1204 13:54:51.945000 341159 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 341229
2025-12-04T13:57:16.2533890Z I1204 13:54:51.945000 341159 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 341230
2025-12-04T13:57:16.2534449Z I1204 13:54:51.946000 341159 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 341231
2025-12-04T13:57:16.2535388Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2536121Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2536912Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2537658Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2538374Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2539084Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2539796Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2540514Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2541360Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2542150Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2542874Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2543871Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2544649Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2545373Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2546094Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2546908Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2547396Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.2547866Z   return func(*args, **kwargs)
2025-12-04T13:57:16.2548308Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2548760Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2549201Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2549644Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2550082Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2550526Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2550962Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2551377Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2551759Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2552130Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2552488Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2552856Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2553213Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2553579Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2553927Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2554292Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2555694Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2557183Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2558664Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2560091Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2561525Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2562944Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2564368Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2565787Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2566248Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2566612Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2567021Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2567445Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2567811Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2568170Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2568528Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2568888Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2569128Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2569483Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2569989Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2570481Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2570971Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2571429Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2571875Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2572351Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2572824Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2573298Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2573772Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2574233Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2574698Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2575176Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2575862Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1997312 on device 0. CUDA driver allocated memory was 2462056448 and is now 4156555264.
2025-12-04T13:57:16.2576464Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2576856Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2577513Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2577996Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2578373Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2578798Z [rank0]:E1204 13:55:05.836000 341228 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.2579049Z dist init r=0, world=4
2025-12-04T13:57:16.2579264Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2579612Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2580110Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2580599Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2581084Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2581538Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2581986Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2582462Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2582933Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2583401Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2583871Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2584332Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2584792Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2585303Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2585944Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1929728 on device 1. CUDA driver allocated memory was 2317352960 and is now 4013948928.
2025-12-04T13:57:16.2586566Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2586966Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2587537Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2588022Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2588398Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2588820Z [rank1]:E1204 13:55:05.853000 341229 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.2589072Z dist init r=1, world=4
2025-12-04T13:57:16.2589284Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2589627Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2590123Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2590610Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2591097Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2591551Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2591999Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2592468Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2592938Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2593406Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2593877Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2594334Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2594825Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2595301Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2595936Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1963520 on device 3. CUDA driver allocated memory was 2250244096 and is now 3946840064.
2025-12-04T13:57:16.2596562Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2596962Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2597526Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2598004Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2598377Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2598799Z [rank3]:E1204 13:55:05.921000 341231 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.2599045Z dist init r=3, world=4
2025-12-04T13:57:16.2599254Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2599598Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2600093Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2600580Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2601062Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2601515Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2601961Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2602441Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2602919Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2603389Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2603899Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2604357Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2604823Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2605334Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2605976Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1929728 on device 2. CUDA driver allocated memory was 2300575744 and is now 3997171712.
2025-12-04T13:57:16.2606568Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2606965Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2607534Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2608021Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2608393Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2608818Z [rank2]:E1204 13:55:05.933000 341230 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.2609067Z dist init r=2, world=4
2025-12-04T13:57:16.2609503Z [rank0]:[W1204 13:55:06.550236306 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.2609923Z FAILED [15.7523s] [ 25%]
2025-12-04T13:57:16.2609994Z 
2025-12-04T13:57:16.2610060Z =================================== FAILURES ===================================
2025-12-04T13:57:16.2610251Z __________________ TestClipGradNormCUDA.test_ddp_parity_cuda ___________________
2025-12-04T13:57:16.2610427Z Traceback (most recent call last):
2025-12-04T13:57:16.2610682Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.2610932Z     self._join_processes(fn)
2025-12-04T13:57:16.2611186Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.2611459Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.2611734Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.2612003Z     raise RuntimeError(error)
2025-12-04T13:57:16.2612163Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.2612331Z Traceback (most recent call last):
2025-12-04T13:57:16.2612576Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2612826Z     getattr(self, test_name)()
2025-12-04T13:57:16.2613099Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2613340Z     fn()
2025-12-04T13:57:16.2613549Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2613786Z     method(*args, **kwargs)
2025-12-04T13:57:16.2614014Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2614249Z     method(*args, **kwargs)
2025-12-04T13:57:16.2614510Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2614742Z     with policy():
2025-12-04T13:57:16.2614962Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2615196Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2615587Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1997312 on device 0. CUDA driver allocated memory was 2462056448 and is now 4156555264.
2025-12-04T13:57:16.2615935Z 
2025-12-04T13:57:16.2616014Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2616330Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2616572Z 
2025-12-04T13:57:16.2616666Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2616836Z 
2025-12-04T13:57:16.2616842Z 
2025-12-04T13:57:16.2616922Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.2617128Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.2617520Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-1c423d878373b38f.xml -
2025-12-04T13:57:16.2617883Z =========================== short test summary info ============================
2025-12-04T13:57:16.2618207Z FAILED [15.7523s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.2618510Z Traceback (most recent call last):
2025-12-04T13:57:16.2618761Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2619013Z     getattr(self, test_name)()
2025-12-04T13:57:16.2619251Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2619489Z     fn()
2025-12-04T13:57:16.2619696Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2619933Z     method(*args, **kwargs)
2025-12-04T13:57:16.2620153Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2620384Z     method(*args, **kwargs)
2025-12-04T13:57:16.2620601Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2620830Z     with policy():
2025-12-04T13:57:16.2621042Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2621272Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2621655Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1997312 on device 0. CUDA driver allocated memory was 2462056448 and is now 4156555264.
2025-12-04T13:57:16.2622040Z 
2025-12-04T13:57:16.2622116Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2622429Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2622667Z 
2025-12-04T13:57:16.2622755Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2622945Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.2623135Z ============================== 1 failed in 15.76s ==============================
2025-12-04T13:57:16.2623269Z Got exit code 1
2025-12-04T13:57:16.2623367Z Retrying single test...
2025-12-04T13:57:16.2623647Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-6bd84d8a44c3f059.xml
2025-12-04T13:57:16.2623958Z ============================= test session starts ==============================
2025-12-04T13:57:16.2624169Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.2624355Z cachedir: .pytest_cache
2025-12-04T13:57:16.2624581Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.2624820Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.2624939Z configfile: pytest.ini
2025-12-04T13:57:16.2625164Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.2625439Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.2625741Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda
2025-12-04T13:57:16.2626011Z Running 1 items in this shard
2025-12-04T13:57:16.2626086Z 
2025-12-04T13:57:16.2626374Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda I1204 13:55:10.345000 342069 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 342138
2025-12-04T13:57:16.2626890Z I1204 13:55:10.346000 342069 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 342139
2025-12-04T13:57:16.2627237Z I1204 13:55:10.346000 342069 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 342140
2025-12-04T13:57:16.2627583Z I1204 13:55:10.347000 342069 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 342141
2025-12-04T13:57:16.2628139Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2628583Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2629021Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2629457Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2629889Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2630336Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2630806Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2631245Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2631824Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2632449Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2633044Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2633636Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2634222Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2634808Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2635398Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2635982Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2636373Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.2636786Z   return func(*args, **kwargs)
2025-12-04T13:57:16.2637141Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2637506Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2637860Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2638220Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2638570Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2638933Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2639291Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2639651Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2640013Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2640376Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2640768Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2641130Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2641475Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2641844Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2642228Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2642589Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2643941Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2645368Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2646834Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2648258Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2649721Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2651153Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2652590Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2654035Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2654471Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2654834Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2655193Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2655556Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2655915Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2656276Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2656636Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2657044Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2657282Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2657633Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2658135Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2658625Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2659115Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2659573Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2660022Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2660525Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2661000Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2661472Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2661980Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2662438Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2662903Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2663377Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2664016Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1997312 on device 1. CUDA driver allocated memory was 2317352960 and is now 4013948928.
2025-12-04T13:57:16.2664615Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2664971Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2665535Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2666014Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2666386Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2666844Z [rank1]:E1204 13:55:24.542000 342139 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.2667092Z dist init r=1, world=4
2025-12-04T13:57:16.2667303Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2667651Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2668146Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2668626Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2669108Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2669562Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2670045Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2670522Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2670989Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2671497Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2671963Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2672420Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2672876Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2673354Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2673986Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1997312 on device 0. CUDA driver allocated memory was 2459959296 and is now 4156555264.
2025-12-04T13:57:16.2674580Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2674933Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2675493Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2675977Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2676343Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2676807Z [rank0]:E1204 13:55:24.558000 342138 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.2677050Z dist init r=0, world=4
2025-12-04T13:57:16.2677254Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2677591Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2678080Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2678562Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2679072Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2679521Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2679963Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2680428Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2680926Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2681392Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2681866Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2682322Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2682787Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2683266Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2683902Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1929728 on device 2. CUDA driver allocated memory was 2300575744 and is now 3997171712.
2025-12-04T13:57:16.2684499Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2684850Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2685411Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2685886Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2686255Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2686668Z [rank2]:E1204 13:55:24.599000 342140 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.2686958Z dist init r=2, world=4
2025-12-04T13:57:16.2687167Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2687510Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2687998Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2688513Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2688993Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2689440Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2689880Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2690380Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2690855Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2691320Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2691789Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2692245Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2692699Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2693174Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2693805Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1963520 on device 3. CUDA driver allocated memory was 2243952640 and is now 3946840064.
2025-12-04T13:57:16.2694393Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2694750Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2695317Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2695802Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2696180Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2696595Z [rank3]:E1204 13:55:24.612000 342141 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.2696887Z dist init r=3, world=4
2025-12-04T13:57:16.2697288Z [rank0]:[W1204 13:55:24.285625587 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.2697697Z FAILED [16.2404s] [100%]
2025-12-04T13:57:16.2697762Z 
2025-12-04T13:57:16.2697851Z =================================== FAILURES ===================================
2025-12-04T13:57:16.2698036Z __________________ TestClipGradNormCUDA.test_ddp_parity_cuda ___________________
2025-12-04T13:57:16.2698208Z Traceback (most recent call last):
2025-12-04T13:57:16.2698452Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.2698697Z     self._join_processes(fn)
2025-12-04T13:57:16.2698945Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.2699242Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.2699514Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.2699777Z     raise RuntimeError(error)
2025-12-04T13:57:16.2699934Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.2700097Z Traceback (most recent call last):
2025-12-04T13:57:16.2700340Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2700584Z     getattr(self, test_name)()
2025-12-04T13:57:16.2700815Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2701050Z     fn()
2025-12-04T13:57:16.2701252Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2701485Z     method(*args, **kwargs)
2025-12-04T13:57:16.2701706Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2701936Z     method(*args, **kwargs)
2025-12-04T13:57:16.2702159Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2702385Z     with policy():
2025-12-04T13:57:16.2702597Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2702827Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2703211Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1997312 on device 1. CUDA driver allocated memory was 2317352960 and is now 4013948928.
2025-12-04T13:57:16.2703560Z 
2025-12-04T13:57:16.2703638Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2703949Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2704184Z 
2025-12-04T13:57:16.2704279Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2704404Z 
2025-12-04T13:57:16.2704408Z 
2025-12-04T13:57:16.2704485Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.2704688Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.2705076Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-6bd84d8a44c3f059.xml -
2025-12-04T13:57:16.2705435Z =========================== short test summary info ============================
2025-12-04T13:57:16.2705756Z FAILED [16.2404s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.2706056Z Traceback (most recent call last):
2025-12-04T13:57:16.2706335Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2706583Z     getattr(self, test_name)()
2025-12-04T13:57:16.2706865Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2707098Z     fn()
2025-12-04T13:57:16.2707304Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2707534Z     method(*args, **kwargs)
2025-12-04T13:57:16.2707797Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2719077Z     method(*args, **kwargs)
2025-12-04T13:57:16.2719336Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2719567Z     with policy():
2025-12-04T13:57:16.2719798Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2720034Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2720427Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1997312 on device 1. CUDA driver allocated memory was 2317352960 and is now 4013948928.
2025-12-04T13:57:16.2720783Z 
2025-12-04T13:57:16.2720859Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2721177Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2721418Z 
2025-12-04T13:57:16.2721510Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2721711Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.2721893Z ======================= 1 failed, 3 deselected in 16.25s =======================
2025-12-04T13:57:16.2722044Z Got exit code 1
2025-12-04T13:57:16.2722149Z Retrying single test...
2025-12-04T13:57:16.2722432Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-968819895df981c7.xml
2025-12-04T13:57:16.2722743Z ============================= test session starts ==============================
2025-12-04T13:57:16.2722959Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.2723154Z cachedir: .pytest_cache
2025-12-04T13:57:16.2723383Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.2723626Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.2723751Z configfile: pytest.ini
2025-12-04T13:57:16.2723986Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.2724261Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.2724566Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda
2025-12-04T13:57:16.2724839Z Running 1 items in this shard
2025-12-04T13:57:16.2724913Z 
2025-12-04T13:57:16.2725198Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda I1204 13:55:29.295000 342979 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 343048
2025-12-04T13:57:16.2725674Z I1204 13:55:29.296000 342979 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 343049
2025-12-04T13:57:16.2726022Z I1204 13:55:29.296000 342979 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 343050
2025-12-04T13:57:16.2726424Z I1204 13:55:29.297000 342979 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 343051
2025-12-04T13:57:16.2727020Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2727465Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2727906Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2728377Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2728820Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2729258Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2729691Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T13:57:16.2730129Z   self.encoder = TransformerEncoder(
2025-12-04T13:57:16.2730709Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2731307Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2731895Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2732480Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2733067Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2733649Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2734233Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2734811Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2735203Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.2735579Z   return func(*args, **kwargs)
2025-12-04T13:57:16.2735979Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2736352Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2736715Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2737122Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2737482Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2737881Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2738238Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2738599Z   return fsdp_fn(module, **kwargs)
2025-12-04T13:57:16.2738974Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2739348Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2739712Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2740089Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2740446Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2740814Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2741170Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2741536Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2742890Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2744330Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2745804Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2747286Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2748759Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2750179Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2751604Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T13:57:16.2753030Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T13:57:16.2753477Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2753851Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2754218Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2754586Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2754952Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2755310Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2755665Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:123: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2756018Z   fsdp_model.transformer.encoder = FSDP(
2025-12-04T13:57:16.2756293Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2756643Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2757182Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2757701Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2758186Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2758644Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2759095Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2759567Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2760038Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2760516Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2760986Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2761443Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2761901Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2762379Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2763028Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1929728 on device 1. CUDA driver allocated memory was 2317352960 and is now 4013948928.
2025-12-04T13:57:16.2763633Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2763996Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2764568Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2765061Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2765440Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2765896Z [rank1]:E1204 13:55:43.278000 343049 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.2766147Z dist init r=1, world=4
2025-12-04T13:57:16.2766361Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2766710Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2767269Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2767759Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2768254Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2768713Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2769165Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2769642Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2770116Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2770593Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2771066Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2771528Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2771994Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2772471Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2773108Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1963520 on device 3. CUDA driver allocated memory was 2250244096 and is now 3946840064.
2025-12-04T13:57:16.2773702Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2774057Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2774621Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2775133Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2775508Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2775925Z [rank3]:E1204 13:55:43.284000 343051 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.2776171Z dist init r=3, world=4
2025-12-04T13:57:16.2776378Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2776781Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2777274Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2777754Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2778232Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2778678Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2779122Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2779593Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2780065Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2780532Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2780999Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2781454Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2781914Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2782383Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2783016Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1963520 on device 0. CUDA driver allocated memory was 2459959296 and is now 4156555264.
2025-12-04T13:57:16.2783631Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2783988Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2784579Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2785059Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2785432Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2785893Z [rank0]:E1204 13:55:43.351000 343048 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.2786142Z dist init r=0, world=4
2025-12-04T13:57:16.2786349Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2786691Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2787223Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2787706Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2788188Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2788644Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2789091Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2789581Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2790049Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2790519Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2790986Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2791442Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2791903Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2792373Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2793008Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1929728 on device 2. CUDA driver allocated memory was 2300575744 and is now 3997171712.
2025-12-04T13:57:16.2793602Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2793986Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2794548Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2795026Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2795428Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2795843Z [rank2]:E1204 13:55:43.354000 343050 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.2796087Z dist init r=2, world=4
2025-12-04T13:57:16.2796492Z [rank0]:[W1204 13:55:43.186151430 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.2796947Z FAILED [16.0426s] [100%]
2025-12-04T13:57:16.2797018Z 
2025-12-04T13:57:16.2797078Z =================================== FAILURES ===================================
2025-12-04T13:57:16.2797267Z __________________ TestClipGradNormCUDA.test_ddp_parity_cuda ___________________
2025-12-04T13:57:16.2797445Z Traceback (most recent call last):
2025-12-04T13:57:16.2797697Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.2797948Z     self._join_processes(fn)
2025-12-04T13:57:16.2798202Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.2798477Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.2798751Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.2799019Z     raise RuntimeError(error)
2025-12-04T13:57:16.2799178Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.2799347Z Traceback (most recent call last):
2025-12-04T13:57:16.2799593Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2799843Z     getattr(self, test_name)()
2025-12-04T13:57:16.2800083Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2800322Z     fn()
2025-12-04T13:57:16.2800533Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2800771Z     method(*args, **kwargs)
2025-12-04T13:57:16.2800998Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2801233Z     method(*args, **kwargs)
2025-12-04T13:57:16.2801457Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2801690Z     with policy():
2025-12-04T13:57:16.2801910Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2802147Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2802535Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1929728 on device 1. CUDA driver allocated memory was 2317352960 and is now 4013948928.
2025-12-04T13:57:16.2802886Z 
2025-12-04T13:57:16.2803001Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2803320Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2803561Z 
2025-12-04T13:57:16.2803653Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2803785Z 
2025-12-04T13:57:16.2803847Z Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.2804028Z Traceback (most recent call last):
2025-12-04T13:57:16.2804277Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2804526Z     getattr(self, test_name)()
2025-12-04T13:57:16.2804762Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2805000Z     fn()
2025-12-04T13:57:16.2805208Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2805442Z     method(*args, **kwargs)
2025-12-04T13:57:16.2805665Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2805898Z     method(*args, **kwargs)
2025-12-04T13:57:16.2806121Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2806355Z     with policy():
2025-12-04T13:57:16.2806572Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2806852Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2807243Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1963520 on device 3. CUDA driver allocated memory was 2250244096 and is now 3946840064.
2025-12-04T13:57:16.2807595Z 
2025-12-04T13:57:16.2807671Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2807986Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2808225Z 
2025-12-04T13:57:16.2808316Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2808449Z 
2025-12-04T13:57:16.2808451Z 
2025-12-04T13:57:16.2808534Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.2808744Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.2809135Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-968819895df981c7.xml -
2025-12-04T13:57:16.2809500Z =========================== short test summary info ============================
2025-12-04T13:57:16.2809826Z FAILED [16.0426s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.2810132Z Traceback (most recent call last):
2025-12-04T13:57:16.2810384Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2810636Z     getattr(self, test_name)()
2025-12-04T13:57:16.2810875Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2811114Z     fn()
2025-12-04T13:57:16.2811322Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2811559Z     method(*args, **kwargs)
2025-12-04T13:57:16.2811816Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2812049Z     method(*args, **kwargs)
2025-12-04T13:57:16.2812269Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2812499Z     with policy():
2025-12-04T13:57:16.2812711Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2812981Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2813365Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1929728 on device 1. CUDA driver allocated memory was 2317352960 and is now 4013948928.
2025-12-04T13:57:16.2813712Z 
2025-12-04T13:57:16.2813790Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2814106Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2814341Z 
2025-12-04T13:57:16.2814433Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2814557Z 
2025-12-04T13:57:16.2814620Z Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.2814767Z Traceback (most recent call last):
2025-12-04T13:57:16.2815012Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2815259Z     getattr(self, test_name)()
2025-12-04T13:57:16.2815494Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2815729Z     fn()
2025-12-04T13:57:16.2815936Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2816169Z     method(*args, **kwargs)
2025-12-04T13:57:16.2816392Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2816621Z     method(*args, **kwargs)
2025-12-04T13:57:16.2816885Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2817115Z     with policy():
2025-12-04T13:57:16.2817330Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2817567Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2817951Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_ddp_parity_cuda! Caching allocator allocated memory was 512 and is now reported as 1963520 on device 3. CUDA driver allocated memory was 2250244096 and is now 3946840064.
2025-12-04T13:57:16.2818299Z 
2025-12-04T13:57:16.2818379Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2818693Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_ddp_parity_cuda
2025-12-04T13:57:16.2818932Z 
2025-12-04T13:57:16.2819021Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2819214Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.2819386Z ======================= 1 failed, 3 deselected in 16.05s =======================
2025-12-04T13:57:16.2819528Z Got exit code 1
2025-12-04T13:57:16.2819740Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda
2025-12-04T13:57:16.2820055Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T13:57:16.2820480Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-5a0a53e51ea4bef7.xml
2025-12-04T13:57:16.2820794Z ============================= test session starts ==============================
2025-12-04T13:57:16.2821009Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.2821202Z cachedir: .pytest_cache
2025-12-04T13:57:16.2821432Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.2821712Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.2821834Z configfile: pytest.ini
2025-12-04T13:57:16.2822067Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.2822342Z collecting ... collected 4 items / 1 deselected / 3 selected
2025-12-04T13:57:16.2822508Z stepcurrent: skipping 1 already run items.
2025-12-04T13:57:16.2822641Z Running 3 items in this shard
2025-12-04T13:57:16.2822718Z 
2025-12-04T13:57:16.2823016Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda I1204 13:55:47.860000 343889 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 343958
2025-12-04T13:57:16.2823500Z I1204 13:55:47.861000 343889 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 343959
2025-12-04T13:57:16.2823852Z I1204 13:55:47.861000 343889 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 343960
2025-12-04T13:57:16.2824199Z I1204 13:55:47.862000 343889 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 343961
2025-12-04T13:57:16.2824900Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2825498Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2826091Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2826686Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2827301Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2827878Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2828460Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2829047Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2829468Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.2829842Z   return func(*args, **kwargs)
2025-12-04T13:57:16.2830210Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2830590Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2830965Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2831374Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2831745Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2832125Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2832498Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2832870Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2833230Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2833581Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2833915Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2834260Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2834593Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2834936Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2835261Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2835598Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2835808Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2836152Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2836646Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2837165Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2837650Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2838104Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2838549Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2839049Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2839522Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2839988Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2840458Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2840946Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2841407Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2841876Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2842521Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2250244096 and is now 3416260608.
2025-12-04T13:57:16.2843125Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2843476Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2844054Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2844549Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2844916Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2845334Z [rank3]:E1204 13:55:53.681000 343961 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.2845580Z dist init r=3, world=4
2025-12-04T13:57:16.2845785Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2846122Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2846610Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2847125Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2847609Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2848065Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2848536Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2848998Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2849463Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2849954Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2850417Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2850869Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2851322Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2851793Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2852436Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 1. CUDA driver allocated memory was 2317352960 and is now 3483369472.
2025-12-04T13:57:16.2853036Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2853385Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2853959Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2854452Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2854822Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2855241Z [rank1]:E1204 13:55:53.688000 343959 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.2855483Z dist init r=1, world=4
2025-12-04T13:57:16.2855685Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2856023Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2856511Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2857030Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2857550Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2857998Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2858436Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2858898Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2859393Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2859854Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2860317Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2860770Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2861220Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2861685Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2862325Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 2. CUDA driver allocated memory was 2300575744 and is now 3466592256.
2025-12-04T13:57:16.2862922Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2863269Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2863839Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2864325Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2864689Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2865098Z [rank2]:E1204 13:55:53.692000 343960 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.2865338Z dist init r=2, world=4
2025-12-04T13:57:16.2865539Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2865875Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2866361Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2866915Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2867395Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2867841Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2868279Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2868770Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2869235Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2869695Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2870153Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2870603Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2871057Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2871521Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2872163Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 0. CUDA driver allocated memory was 2459959296 and is now 3625975808.
2025-12-04T13:57:16.2872757Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2873107Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2873678Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2874167Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2874530Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2874943Z [rank0]:E1204 13:55:53.696000 343958 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.2875185Z dist init r=0, world=4
2025-12-04T13:57:16.2875583Z [rank0]:[W1204 13:55:53.425408848 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.2875993Z FAILED [7.5127s] [ 33%]
2025-12-04T13:57:16.2876057Z 
2025-12-04T13:57:16.2876145Z =================================== FAILURES ===================================
2025-12-04T13:57:16.2876333Z ______________ TestClipGradNormCUDA.test_low_precision_grads_cuda ______________
2025-12-04T13:57:16.2876505Z Traceback (most recent call last):
2025-12-04T13:57:16.2876790Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.2877034Z     self._join_processes(fn)
2025-12-04T13:57:16.2877280Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.2877578Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.2877844Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.2878102Z     raise RuntimeError(error)
2025-12-04T13:57:16.2878253Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.2878420Z Traceback (most recent call last):
2025-12-04T13:57:16.2878660Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2878900Z     getattr(self, test_name)()
2025-12-04T13:57:16.2879130Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2879361Z     fn()
2025-12-04T13:57:16.2879565Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2879793Z     method(*args, **kwargs)
2025-12-04T13:57:16.2880012Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2880239Z     method(*args, **kwargs)
2025-12-04T13:57:16.2880458Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2880683Z     with policy():
2025-12-04T13:57:16.2880892Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2881121Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2881509Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2250244096 and is now 3416260608.
2025-12-04T13:57:16.2881866Z 
2025-12-04T13:57:16.2881941Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2882261Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2882506Z 
2025-12-04T13:57:16.2882596Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2882718Z 
2025-12-04T13:57:16.2882720Z 
2025-12-04T13:57:16.2882798Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.2882997Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.2883380Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-5a0a53e51ea4bef7.xml -
2025-12-04T13:57:16.2883733Z =========================== short test summary info ============================
2025-12-04T13:57:16.2884058Z FAILED [7.5127s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.2884365Z Traceback (most recent call last):
2025-12-04T13:57:16.2884637Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2884878Z     getattr(self, test_name)()
2025-12-04T13:57:16.2885107Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2885336Z     fn()
2025-12-04T13:57:16.2885534Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2885759Z     method(*args, **kwargs)
2025-12-04T13:57:16.2886000Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2886225Z     method(*args, **kwargs)
2025-12-04T13:57:16.2886438Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2886659Z     with policy():
2025-12-04T13:57:16.2886908Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2887136Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2887527Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2250244096 and is now 3416260608.
2025-12-04T13:57:16.2887883Z 
2025-12-04T13:57:16.2887957Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2888280Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2888527Z 
2025-12-04T13:57:16.2888614Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2888800Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.2888966Z ======================= 1 failed, 1 deselected in 7.52s ========================
2025-12-04T13:57:16.2889102Z Got exit code 1
2025-12-04T13:57:16.2889197Z Retrying single test...
2025-12-04T13:57:16.2889473Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-6bb9b908fdb329d1.xml
2025-12-04T13:57:16.2889778Z ============================= test session starts ==============================
2025-12-04T13:57:16.2889985Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.2890173Z cachedir: .pytest_cache
2025-12-04T13:57:16.2890393Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.2890630Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.2890745Z configfile: pytest.ini
2025-12-04T13:57:16.2890973Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.2891238Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.2891547Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda
2025-12-04T13:57:16.2891825Z Running 1 items in this shard
2025-12-04T13:57:16.2891897Z 
2025-12-04T13:57:16.2892191Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda I1204 13:55:57.682000 344291 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 344360
2025-12-04T13:57:16.2892671Z I1204 13:55:57.683000 344291 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 344361
2025-12-04T13:57:16.2893017Z I1204 13:55:57.684000 344291 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 344362
2025-12-04T13:57:16.2893397Z I1204 13:55:57.684000 344291 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 344363
2025-12-04T13:57:16.2894083Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2894700Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2895284Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2895865Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2896447Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2897060Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2897645Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2898224Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2898614Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.2898982Z   return func(*args, **kwargs)
2025-12-04T13:57:16.2899343Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2899717Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2900086Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2900452Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2900817Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2901185Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2901551Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2901919Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2902277Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2902621Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2902987Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2903330Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2903657Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2904030Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2904354Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2904691Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2904894Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2905239Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2905731Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2906212Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2906691Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2907194Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2907639Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2908102Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2908566Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2909028Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2909491Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2909940Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2910393Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2910857Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2911500Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 1. CUDA driver allocated memory was 2317352960 and is now 3483369472.
2025-12-04T13:57:16.2912128Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2912477Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2913049Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2913570Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2913939Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2914357Z [rank1]:E1204 13:56:03.345000 344361 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.2914700Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2915038Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2915526Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2916008Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2916488Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2916969Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2917412Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2917877Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2918346Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2918809Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2919279Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2919734Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2920194Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2920661Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2921339Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 2. CUDA driver allocated memory was 2300575744 and is now 3466592256.
2025-12-04T13:57:16.2921938Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2922290Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2922908Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2923397Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2923766Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2924182Z [rank2]:E1204 13:56:03.345000 344362 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.2924428Z dist init r=2, world=4
2025-12-04T13:57:16.2924535Z dist init r=1, world=4
2025-12-04T13:57:16.2924736Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2925078Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2925568Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2926047Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2926526Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2927017Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2927460Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2927927Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2928392Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2928856Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2929319Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2929771Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2930225Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2930721Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2931357Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 0. CUDA driver allocated memory was 2459959296 and is now 3625975808.
2025-12-04T13:57:16.2931985Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2932336Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2932909Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2933402Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2933769Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2934188Z [rank0]:E1204 13:56:03.347000 344360 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.2934434Z dist init r=0, world=4
2025-12-04T13:57:16.2934639Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2934979Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2935468Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2935948Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2936429Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2936924Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2937372Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2937836Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2938300Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2938764Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2939233Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2939683Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2940169Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2940639Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2941278Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2243952640 and is now 3416260608.
2025-12-04T13:57:16.2941908Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2942263Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2942834Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2943319Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2943685Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2944100Z [rank3]:E1204 13:56:03.356000 344363 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.2944348Z dist init r=3, world=4
2025-12-04T13:57:16.2944750Z [rank0]:[W1204 13:56:03.013129721 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.2945159Z FAILED [7.3121s] [100%]
2025-12-04T13:57:16.2945225Z 
2025-12-04T13:57:16.2945284Z =================================== FAILURES ===================================
2025-12-04T13:57:16.2945472Z ______________ TestClipGradNormCUDA.test_low_precision_grads_cuda ______________
2025-12-04T13:57:16.2945648Z Traceback (most recent call last):
2025-12-04T13:57:16.2945894Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.2946138Z     self._join_processes(fn)
2025-12-04T13:57:16.2946386Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.2946654Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.2946976Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.2947236Z     raise RuntimeError(error)
2025-12-04T13:57:16.2947391Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.2947552Z Traceback (most recent call last):
2025-12-04T13:57:16.2947794Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2948038Z     getattr(self, test_name)()
2025-12-04T13:57:16.2948270Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2948502Z     fn()
2025-12-04T13:57:16.2948706Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2948969Z     method(*args, **kwargs)
2025-12-04T13:57:16.2949193Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2949424Z     method(*args, **kwargs)
2025-12-04T13:57:16.2949643Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2949871Z     with policy():
2025-12-04T13:57:16.2950084Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2950347Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2950744Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 0. CUDA driver allocated memory was 2459959296 and is now 3625975808.
2025-12-04T13:57:16.2951108Z 
2025-12-04T13:57:16.2951186Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2951512Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2951767Z 
2025-12-04T13:57:16.2951856Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2951987Z 
2025-12-04T13:57:16.2952049Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.2952194Z Traceback (most recent call last):
2025-12-04T13:57:16.2952439Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2952683Z     getattr(self, test_name)()
2025-12-04T13:57:16.2952919Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2953152Z     fn()
2025-12-04T13:57:16.2953356Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2953587Z     method(*args, **kwargs)
2025-12-04T13:57:16.2953804Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2954035Z     method(*args, **kwargs)
2025-12-04T13:57:16.2954250Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2954485Z     with policy():
2025-12-04T13:57:16.2954692Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2954924Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2955323Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 2. CUDA driver allocated memory was 2300575744 and is now 3466592256.
2025-12-04T13:57:16.2955682Z 
2025-12-04T13:57:16.2955760Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2956082Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2956329Z 
2025-12-04T13:57:16.2956419Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2956545Z 
2025-12-04T13:57:16.2956606Z Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.2956783Z Traceback (most recent call last):
2025-12-04T13:57:16.2957025Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2957067Z     getattr(self, test_name)()
2025-12-04T13:57:16.2957263Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2957299Z     fn()
2025-12-04T13:57:16.2957454Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2957494Z     method(*args, **kwargs)
2025-12-04T13:57:16.2957646Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2957686Z     method(*args, **kwargs)
2025-12-04T13:57:16.2957876Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2957913Z     with policy():
2025-12-04T13:57:16.2958068Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2958111Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2958441Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2243952640 and is now 3416260608.
2025-12-04T13:57:16.2958443Z 
2025-12-04T13:57:16.2958520Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2958730Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2958734Z 
2025-12-04T13:57:16.2958825Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2958827Z 
2025-12-04T13:57:16.2958829Z 
2025-12-04T13:57:16.2958906Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.2958998Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.2959263Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-6bb9b908fdb329d1.xml -
2025-12-04T13:57:16.2959327Z =========================== short test summary info ============================
2025-12-04T13:57:16.2959562Z FAILED [7.3121s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.2959610Z Traceback (most recent call last):
2025-12-04T13:57:16.2959781Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2959824Z     getattr(self, test_name)()
2025-12-04T13:57:16.2959990Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2960023Z     fn()
2025-12-04T13:57:16.2960177Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2960217Z     method(*args, **kwargs)
2025-12-04T13:57:16.2960372Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2960412Z     method(*args, **kwargs)
2025-12-04T13:57:16.2960563Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2960600Z     with policy():
2025-12-04T13:57:16.2960755Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2960798Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2961122Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 0. CUDA driver allocated memory was 2459959296 and is now 3625975808.
2025-12-04T13:57:16.2961147Z 
2025-12-04T13:57:16.2961222Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2961431Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2961433Z 
2025-12-04T13:57:16.2961521Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2961523Z 
2025-12-04T13:57:16.2961582Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.2961653Z Traceback (most recent call last):
2025-12-04T13:57:16.2961816Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2961859Z     getattr(self, test_name)()
2025-12-04T13:57:16.2962017Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2962054Z     fn()
2025-12-04T13:57:16.2962203Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2962243Z     method(*args, **kwargs)
2025-12-04T13:57:16.2962393Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2962432Z     method(*args, **kwargs)
2025-12-04T13:57:16.2962580Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2962620Z     with policy():
2025-12-04T13:57:16.2962770Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2962811Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2963131Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 2. CUDA driver allocated memory was 2300575744 and is now 3466592256.
2025-12-04T13:57:16.2963134Z 
2025-12-04T13:57:16.2963206Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2963415Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2963418Z 
2025-12-04T13:57:16.2963504Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2963508Z 
2025-12-04T13:57:16.2963567Z Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.2963612Z Traceback (most recent call last):
2025-12-04T13:57:16.2963774Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2963815Z     getattr(self, test_name)()
2025-12-04T13:57:16.2963976Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2964010Z     fn()
2025-12-04T13:57:16.2964161Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2964200Z     method(*args, **kwargs)
2025-12-04T13:57:16.2964349Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2964390Z     method(*args, **kwargs)
2025-12-04T13:57:16.2964539Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2964575Z     with policy():
2025-12-04T13:57:16.2964727Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2964767Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2965109Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2243952640 and is now 3416260608.
2025-12-04T13:57:16.2965112Z 
2025-12-04T13:57:16.2965187Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2965394Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2965420Z 
2025-12-04T13:57:16.2965507Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2965571Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.2965637Z ======================= 1 failed, 3 deselected in 7.32s ========================
2025-12-04T13:57:16.2965673Z Got exit code 1
2025-12-04T13:57:16.2965716Z Retrying single test...
2025-12-04T13:57:16.2965931Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-ea36288bd7152c54.xml
2025-12-04T13:57:16.2965991Z ============================= test session starts ==============================
2025-12-04T13:57:16.2966105Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.2966148Z cachedir: .pytest_cache
2025-12-04T13:57:16.2966310Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.2966359Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.2966398Z configfile: pytest.ini
2025-12-04T13:57:16.2966563Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.2966635Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.2966886Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda
2025-12-04T13:57:16.2966931Z Running 1 items in this shard
2025-12-04T13:57:16.2966933Z 
2025-12-04T13:57:16.2967223Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda I1204 13:56:07.252000 344693 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 344762
2025-12-04T13:57:16.2967382Z I1204 13:56:07.253000 344693 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 344763
2025-12-04T13:57:16.2967536Z I1204 13:56:07.253000 344693 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 344764
2025-12-04T13:57:16.2967689Z I1204 13:56:07.254000 344693 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 344765
2025-12-04T13:57:16.2968190Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2968253Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2968747Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2968807Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2972998Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2973069Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2973566Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.2973666Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.2973967Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.2974012Z   return func(*args, **kwargs)
2025-12-04T13:57:16.2974302Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2974356Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2974641Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2974690Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2974977Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2975025Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2975307Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2975357Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T13:57:16.2975630Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2975672Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2975949Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2975988Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2976258Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2976297Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2976568Z /var/lib/jenkins/pytorch/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:275: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T13:57:16.2976607Z   fsdp_model = FSDP(
2025-12-04T13:57:16.2976794Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2976958Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2977293Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2977449Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2977737Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2977894Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2978175Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2978326Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2978604Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2978753Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2979030Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2979169Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2979460Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2979608Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2980064Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 1. CUDA driver allocated memory was 2317352960 and is now 3483369472.
2025-12-04T13:57:16.2980182Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2980383Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2980724Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2980839Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2981055Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2981222Z [rank1]:E1204 13:56:12.903000 344763 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.2981263Z dist init r=1, world=4
2025-12-04T13:57:16.2981425Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2981586Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2981877Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2982056Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2982341Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2982469Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2982746Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2982896Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2983172Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2983319Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2983596Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2983732Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2984013Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2984166Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2984617Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2250244096 and is now 3416260608.
2025-12-04T13:57:16.2984732Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2984928Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2985266Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2985385Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2985619Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2985786Z [rank3]:E1204 13:56:12.907000 344765 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.2985824Z dist init r=3, world=4
2025-12-04T13:57:16.2985962Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2986121Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2986433Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2986587Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2986915Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2987039Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2987317Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2987468Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2987743Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2987892Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2988168Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2988308Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2988591Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2988742Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2989193Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 2. CUDA driver allocated memory was 2300575744 and is now 3466592256.
2025-12-04T13:57:16.2989307Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2989502Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2989839Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2989982Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2990194Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2990359Z [rank2]:E1204 13:56:12.910000 344764 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.2990401Z dist init r=2, world=4
2025-12-04T13:57:16.2990565Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.2990726Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.2991016Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2991176Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.2991462Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2991591Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.2991868Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2992017Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2992297Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2992444Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.2992723Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2992859Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.2993144Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2993293Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.2993743Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 0. CUDA driver allocated memory was 2459959296 and is now 3625975808.
2025-12-04T13:57:16.2993861Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2994057Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2994426Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2994540Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.2994752Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2994936Z [rank0]:E1204 13:56:12.964000 344762 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.2994977Z dist init r=0, world=4
2025-12-04T13:57:16.2995317Z [rank0]:[W1204 13:56:13.732464000 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.2995361Z FAILED [7.3130s] [100%]
2025-12-04T13:57:16.2995364Z 
2025-12-04T13:57:16.2995430Z =================================== FAILURES ===================================
2025-12-04T13:57:16.2995524Z ______________ TestClipGradNormCUDA.test_low_precision_grads_cuda ______________
2025-12-04T13:57:16.2995576Z Traceback (most recent call last):
2025-12-04T13:57:16.2995742Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.2995788Z     self._join_processes(fn)
2025-12-04T13:57:16.2995962Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.2996019Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.2996200Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.2996246Z     raise RuntimeError(error)
2025-12-04T13:57:16.2996327Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.2996374Z Traceback (most recent call last):
2025-12-04T13:57:16.2996534Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2996578Z     getattr(self, test_name)()
2025-12-04T13:57:16.2996772Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2996816Z     fn()
2025-12-04T13:57:16.2996968Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2997013Z     method(*args, **kwargs)
2025-12-04T13:57:16.2997164Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2997211Z     method(*args, **kwargs)
2025-12-04T13:57:16.2997362Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2997402Z     with policy():
2025-12-04T13:57:16.2997553Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2997598Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2997930Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 1. CUDA driver allocated memory was 2317352960 and is now 3483369472.
2025-12-04T13:57:16.2997933Z 
2025-12-04T13:57:16.2998010Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.2998253Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.2998256Z 
2025-12-04T13:57:16.2998345Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.2998348Z 
2025-12-04T13:57:16.2998410Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.2998456Z Traceback (most recent call last):
2025-12-04T13:57:16.2998625Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.2998699Z     getattr(self, test_name)()
2025-12-04T13:57:16.2998860Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.2998897Z     fn()
2025-12-04T13:57:16.2999049Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2999088Z     method(*args, **kwargs)
2025-12-04T13:57:16.2999239Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.2999278Z     method(*args, **kwargs)
2025-12-04T13:57:16.2999430Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.2999468Z     with policy():
2025-12-04T13:57:16.2999617Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.2999664Z     raise RuntimeError(msg)
2025-12-04T13:57:16.2999985Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 2. CUDA driver allocated memory was 2300575744 and is now 3466592256.
2025-12-04T13:57:16.2999987Z 
2025-12-04T13:57:16.3000063Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3000273Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.3000275Z 
2025-12-04T13:57:16.3000363Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3000365Z 
2025-12-04T13:57:16.3000424Z Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.3000470Z Traceback (most recent call last):
2025-12-04T13:57:16.3000631Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3000674Z     getattr(self, test_name)()
2025-12-04T13:57:16.3000831Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3000866Z     fn()
2025-12-04T13:57:16.3001017Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3001056Z     method(*args, **kwargs)
2025-12-04T13:57:16.3001205Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3001245Z     method(*args, **kwargs)
2025-12-04T13:57:16.3001394Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3001432Z     with policy():
2025-12-04T13:57:16.3001583Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3001625Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3001969Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2250244096 and is now 3416260608.
2025-12-04T13:57:16.3001972Z 
2025-12-04T13:57:16.3002045Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3002255Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.3002257Z 
2025-12-04T13:57:16.3002344Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3002348Z 
2025-12-04T13:57:16.3002372Z 
2025-12-04T13:57:16.3002450Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.3002539Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.3002803Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-ea36288bd7152c54.xml -
2025-12-04T13:57:16.3002870Z =========================== short test summary info ============================
2025-12-04T13:57:16.3003100Z FAILED [7.3130s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.3003151Z Traceback (most recent call last):
2025-12-04T13:57:16.3003316Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3003361Z     getattr(self, test_name)()
2025-12-04T13:57:16.3003520Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3003558Z     fn()
2025-12-04T13:57:16.3003708Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3003749Z     method(*args, **kwargs)
2025-12-04T13:57:16.3003899Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3003941Z     method(*args, **kwargs)
2025-12-04T13:57:16.3004088Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3004125Z     with policy():
2025-12-04T13:57:16.3004276Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3004324Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3004649Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 1. CUDA driver allocated memory was 2317352960 and is now 3483369472.
2025-12-04T13:57:16.3004651Z 
2025-12-04T13:57:16.3004724Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3004935Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.3004938Z 
2025-12-04T13:57:16.3005025Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3005027Z 
2025-12-04T13:57:16.3005088Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.3005133Z Traceback (most recent call last):
2025-12-04T13:57:16.3005295Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3005338Z     getattr(self, test_name)()
2025-12-04T13:57:16.3005496Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3005531Z     fn()
2025-12-04T13:57:16.3005706Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3005748Z     method(*args, **kwargs)
2025-12-04T13:57:16.3005900Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3005939Z     method(*args, **kwargs)
2025-12-04T13:57:16.3006090Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3006127Z     with policy():
2025-12-04T13:57:16.3006278Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3006351Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3006672Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 2. CUDA driver allocated memory was 2300575744 and is now 3466592256.
2025-12-04T13:57:16.3006674Z 
2025-12-04T13:57:16.3006779Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3006992Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.3006994Z 
2025-12-04T13:57:16.3007087Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3007089Z 
2025-12-04T13:57:16.3007147Z Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.3007197Z Traceback (most recent call last):
2025-12-04T13:57:16.3007359Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3007402Z     getattr(self, test_name)()
2025-12-04T13:57:16.3007558Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3007593Z     fn()
2025-12-04T13:57:16.3007742Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3007784Z     method(*args, **kwargs)
2025-12-04T13:57:16.3007935Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3007974Z     method(*args, **kwargs)
2025-12-04T13:57:16.3008123Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3008161Z     with policy():
2025-12-04T13:57:16.3008314Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3008354Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3008677Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_low_precision_grads_cuda! Caching allocator allocated memory was 512 and is now reported as 92672 on device 3. CUDA driver allocated memory was 2250244096 and is now 3416260608.
2025-12-04T13:57:16.3008680Z 
2025-12-04T13:57:16.3008751Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3008960Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_low_precision_grads_cuda
2025-12-04T13:57:16.3008962Z 
2025-12-04T13:57:16.3009049Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3009116Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.3009181Z ======================= 1 failed, 3 deselected in 7.32s ========================
2025-12-04T13:57:16.3009220Z Got exit code 1
2025-12-04T13:57:16.3009382Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda
2025-12-04T13:57:16.3009541Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T13:57:16.3009758Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-8603312bd03c5a6d.xml
2025-12-04T13:57:16.3009818Z ============================= test session starts ==============================
2025-12-04T13:57:16.3009935Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.3010005Z cachedir: .pytest_cache
2025-12-04T13:57:16.3010164Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.3010211Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.3010253Z configfile: pytest.ini
2025-12-04T13:57:16.3010416Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.3010489Z collecting ... collected 4 items / 2 deselected / 2 selected
2025-12-04T13:57:16.3010542Z stepcurrent: skipping 2 already run items.
2025-12-04T13:57:16.3010586Z Running 2 items in this shard
2025-12-04T13:57:16.3010588Z 
2025-12-04T13:57:16.3010871Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda I1204 13:56:16.928000 345095 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 345164
2025-12-04T13:57:16.3011026Z I1204 13:56:16.929000 345095 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 345165
2025-12-04T13:57:16.3011180Z I1204 13:56:16.929000 345095 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 345166
2025-12-04T13:57:16.3011333Z I1204 13:56:16.930000 345095 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 345167
2025-12-04T13:57:16.3011835Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3011898Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3012389Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3012451Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3012938Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3012996Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3013481Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3013541Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3013856Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.3013901Z   return func(*args, **kwargs)
2025-12-04T13:57:16.3014386Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3014466Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3014951Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3015008Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3015493Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3015552Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3016042Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3016100Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3016389Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.3016432Z   return func(*args, **kwargs)
2025-12-04T13:57:16.3016577Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3016779Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3017070Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3017226Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3017510Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3017638Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3017918Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3018092Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3018376Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3018526Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3018804Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3018966Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3019247Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3019398Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3019838Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2459959296 and is now 3114270720.
2025-12-04T13:57:16.3019956Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3020154Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3020486Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3020601Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3020818Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3020986Z [rank0]:E1204 13:56:21.779000 345164 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.3021026Z dist init r=0, world=4
2025-12-04T13:57:16.3021164Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3021324Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3021612Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3021767Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3022054Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3022178Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3022475Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3022624Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3022900Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3023075Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3023350Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3023488Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3023764Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3023914Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3024353Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 1. CUDA driver allocated memory was 2317352960 and is now 2971664384.
2025-12-04T13:57:16.3024469Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3024665Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3024991Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3025107Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3025325Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3025493Z [rank1]:E1204 13:56:21.787000 345165 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.3025534Z dist init r=1, world=4
2025-12-04T13:57:16.3025670Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3025830Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3026116Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3026272Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3026575Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3026701Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3027013Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3027164Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3027474Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3027624Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3027903Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3028038Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3028316Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3028465Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3028903Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 3. CUDA driver allocated memory was 2250244096 and is now 2904555520.
2025-12-04T13:57:16.3029018Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3029212Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3029543Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3029657Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3029873Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3030036Z [rank3]:E1204 13:56:21.855000 345167 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.3030077Z dist init r=3, world=4
2025-12-04T13:57:16.3030215Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3030375Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3030662Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3030846Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3031131Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3031258Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3031537Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3031706Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3031988Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3032135Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3032413Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3032549Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3032828Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3032979Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3033413Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 2. CUDA driver allocated memory was 2300575744 and is now 2954887168.
2025-12-04T13:57:16.3033529Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3033726Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3034057Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3034170Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3034383Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3034549Z [rank2]:E1204 13:56:21.860000 345166 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.3034589Z dist init r=2, world=4
2025-12-04T13:57:16.3034630Z FAILED [5.7107s] [ 50%]
2025-12-04T13:57:16.3034632Z 
2025-12-04T13:57:16.3034690Z =================================== FAILURES ===================================
2025-12-04T13:57:16.3034780Z _________________ TestClipGradNormCUDA.test_no_gradients_cuda __________________
2025-12-04T13:57:16.3034828Z Traceback (most recent call last):
2025-12-04T13:57:16.3035012Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.3035057Z     self._join_processes(fn)
2025-12-04T13:57:16.3035231Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.3035286Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.3035468Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.3035541Z     raise RuntimeError(error)
2025-12-04T13:57:16.3035624Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.3035668Z Traceback (most recent call last):
2025-12-04T13:57:16.3035830Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3035871Z     getattr(self, test_name)()
2025-12-04T13:57:16.3036031Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3036067Z     fn()
2025-12-04T13:57:16.3036220Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3036260Z     method(*args, **kwargs)
2025-12-04T13:57:16.3036413Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3036456Z     method(*args, **kwargs)
2025-12-04T13:57:16.3036609Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3036647Z     with policy():
2025-12-04T13:57:16.3036842Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3036887Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3037198Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2459959296 and is now 3114270720.
2025-12-04T13:57:16.3037201Z 
2025-12-04T13:57:16.3037279Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3037480Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3037484Z 
2025-12-04T13:57:16.3037575Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3037577Z 
2025-12-04T13:57:16.3037579Z 
2025-12-04T13:57:16.3037656Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.3037747Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.3038011Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-8603312bd03c5a6d.xml -
2025-12-04T13:57:16.3038075Z =========================== short test summary info ============================
2025-12-04T13:57:16.3038296Z FAILED [5.7107s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.3038344Z Traceback (most recent call last):
2025-12-04T13:57:16.3038509Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3038551Z     getattr(self, test_name)()
2025-12-04T13:57:16.3038713Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3038746Z     fn()
2025-12-04T13:57:16.3038955Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3038995Z     method(*args, **kwargs)
2025-12-04T13:57:16.3039148Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3039187Z     method(*args, **kwargs)
2025-12-04T13:57:16.3039339Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3039410Z     with policy():
2025-12-04T13:57:16.3039563Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3039603Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3039915Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2459959296 and is now 3114270720.
2025-12-04T13:57:16.3039917Z 
2025-12-04T13:57:16.3039991Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3040195Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3040197Z 
2025-12-04T13:57:16.3040286Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3040351Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.3040414Z ======================= 1 failed, 2 deselected in 5.72s ========================
2025-12-04T13:57:16.3040450Z Got exit code 1
2025-12-04T13:57:16.3040491Z Retrying single test...
2025-12-04T13:57:16.3040704Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-a701ca36f890c243.xml
2025-12-04T13:57:16.3040767Z ============================= test session starts ==============================
2025-12-04T13:57:16.3040880Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.3040925Z cachedir: .pytest_cache
2025-12-04T13:57:16.3041083Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.3041133Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.3041175Z configfile: pytest.ini
2025-12-04T13:57:16.3041341Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.3041413Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.3041611Z stepcurrent: skipping 2 already run items. Running only test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda
2025-12-04T13:57:16.3041659Z Running 1 items in this shard
2025-12-04T13:57:16.3041661Z 
2025-12-04T13:57:16.3041945Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda I1204 13:56:25.227000 345473 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 345542
2025-12-04T13:57:16.3042102Z I1204 13:56:25.228000 345473 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 345543
2025-12-04T13:57:16.3042255Z I1204 13:56:25.228000 345473 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 345544
2025-12-04T13:57:16.3042410Z I1204 13:56:25.229000 345473 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 345545
2025-12-04T13:57:16.3042934Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3043000Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3043487Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3043571Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3044057Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3044116Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3044600Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3044659Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3044953Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.3044996Z   return func(*args, **kwargs)
2025-12-04T13:57:16.3045489Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3045551Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3046033Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3046094Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3046573Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3046632Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3047163Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3047252Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3047545Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.3047587Z   return func(*args, **kwargs)
2025-12-04T13:57:16.3047732Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3047920Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3048210Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3048368Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3048650Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3048777Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3049054Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3049204Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3049481Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3049630Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3049909Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3050047Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3050330Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3050479Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3050915Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 2. CUDA driver allocated memory was 2300575744 and is now 2954887168.
2025-12-04T13:57:16.3051032Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3051229Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3051581Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3051696Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3051907Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3052073Z [rank2]:E1204 13:56:30.065000 345544 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.3052136Z dist init r=2, world=4
2025-12-04T13:57:16.3052273Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3052432Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3052720Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3052877Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3053161Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3053286Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3053563Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3053711Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3053988Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3054133Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3054415Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3054551Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3054829Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3054979Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3055414Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 3. CUDA driver allocated memory was 2250244096 and is now 2904555520.
2025-12-04T13:57:16.3055532Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3055754Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3056080Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3056194Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3056426Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3056591Z [rank3]:E1204 13:56:30.074000 345545 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.3056629Z dist init r=3, world=4
2025-12-04T13:57:16.3056803Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3056962Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3057250Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3057403Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3057690Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3057817Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3058092Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3058239Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3058516Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3058665Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3058944Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3059084Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3059361Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3059512Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3059949Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2459959296 and is now 3114270720.
2025-12-04T13:57:16.3060097Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3060293Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3060617Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3060765Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3060977Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3061142Z [rank0]:E1204 13:56:30.080000 345542 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.3061182Z dist init r=0, world=4
2025-12-04T13:57:16.3061320Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3061481Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3061768Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3061926Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3062210Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3062335Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3062611Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3062760Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3063037Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3063183Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3063460Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3063596Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3063877Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3064025Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3064481Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 1. CUDA driver allocated memory was 2317352960 and is now 2971664384.
2025-12-04T13:57:16.3064597Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3064793Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3065142Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3065253Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3065467Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3065631Z [rank1]:E1204 13:56:30.081000 345543 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.3065669Z dist init r=1, world=4
2025-12-04T13:57:16.3065708Z FAILED [5.9099s] [100%]
2025-12-04T13:57:16.3065710Z 
2025-12-04T13:57:16.3065767Z =================================== FAILURES ===================================
2025-12-04T13:57:16.3065858Z _________________ TestClipGradNormCUDA.test_no_gradients_cuda __________________
2025-12-04T13:57:16.3065906Z Traceback (most recent call last):
2025-12-04T13:57:16.3066069Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.3066112Z     self._join_processes(fn)
2025-12-04T13:57:16.3066287Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.3066341Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.3066519Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.3066562Z     raise RuntimeError(error)
2025-12-04T13:57:16.3066643Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.3066689Z Traceback (most recent call last):
2025-12-04T13:57:16.3066888Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3066930Z     getattr(self, test_name)()
2025-12-04T13:57:16.3067089Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3067124Z     fn()
2025-12-04T13:57:16.3067276Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3067317Z     method(*args, **kwargs)
2025-12-04T13:57:16.3067469Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3067508Z     method(*args, **kwargs)
2025-12-04T13:57:16.3067660Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3067699Z     with policy():
2025-12-04T13:57:16.3067852Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3067893Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3068237Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2459959296 and is now 3114270720.
2025-12-04T13:57:16.3068239Z 
2025-12-04T13:57:16.3068319Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3068517Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3068519Z 
2025-12-04T13:57:16.3068609Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3068643Z 
2025-12-04T13:57:16.3068702Z Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.3068748Z Traceback (most recent call last):
2025-12-04T13:57:16.3068907Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3068951Z     getattr(self, test_name)()
2025-12-04T13:57:16.3069110Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3069148Z     fn()
2025-12-04T13:57:16.3069298Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3069339Z     method(*args, **kwargs)
2025-12-04T13:57:16.3069489Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3069531Z     method(*args, **kwargs)
2025-12-04T13:57:16.3069680Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3069721Z     with policy():
2025-12-04T13:57:16.3069873Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3069913Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3070227Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 1. CUDA driver allocated memory was 2317352960 and is now 2971664384.
2025-12-04T13:57:16.3070230Z 
2025-12-04T13:57:16.3070304Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3070504Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3070508Z 
2025-12-04T13:57:16.3070595Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3070597Z 
2025-12-04T13:57:16.3070656Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.3070701Z Traceback (most recent call last):
2025-12-04T13:57:16.3070866Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3070909Z     getattr(self, test_name)()
2025-12-04T13:57:16.3071068Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3071102Z     fn()
2025-12-04T13:57:16.3071255Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3071295Z     method(*args, **kwargs)
2025-12-04T13:57:16.3071444Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3071487Z     method(*args, **kwargs)
2025-12-04T13:57:16.3071636Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3071673Z     with policy():
2025-12-04T13:57:16.3071848Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3071894Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3072201Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 2. CUDA driver allocated memory was 2300575744 and is now 2954887168.
2025-12-04T13:57:16.3072203Z 
2025-12-04T13:57:16.3072279Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3072476Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3072500Z 
2025-12-04T13:57:16.3072589Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3072591Z 
2025-12-04T13:57:16.3072593Z 
2025-12-04T13:57:16.3072670Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.3072762Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.3073025Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-a701ca36f890c243.xml -
2025-12-04T13:57:16.3073085Z =========================== short test summary info ============================
2025-12-04T13:57:16.3073306Z FAILED [5.9099s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.3073354Z Traceback (most recent call last):
2025-12-04T13:57:16.3073519Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3073561Z     getattr(self, test_name)()
2025-12-04T13:57:16.3073720Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3073756Z     fn()
2025-12-04T13:57:16.3073911Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3073949Z     method(*args, **kwargs)
2025-12-04T13:57:16.3074100Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3074140Z     method(*args, **kwargs)
2025-12-04T13:57:16.3074291Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3074328Z     with policy():
2025-12-04T13:57:16.3074481Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3074521Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3074838Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2459959296 and is now 3114270720.
2025-12-04T13:57:16.3074840Z 
2025-12-04T13:57:16.3074916Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3075113Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3075115Z 
2025-12-04T13:57:16.3075202Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3075206Z 
2025-12-04T13:57:16.3075266Z Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.3075314Z Traceback (most recent call last):
2025-12-04T13:57:16.3075474Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3075519Z     getattr(self, test_name)()
2025-12-04T13:57:16.3075696Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3075733Z     fn()
2025-12-04T13:57:16.3075884Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3075924Z     method(*args, **kwargs)
2025-12-04T13:57:16.3076072Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3076136Z     method(*args, **kwargs)
2025-12-04T13:57:16.3076286Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3076326Z     with policy():
2025-12-04T13:57:16.3076475Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3076517Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3076886Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 1. CUDA driver allocated memory was 2317352960 and is now 2971664384.
2025-12-04T13:57:16.3076888Z 
2025-12-04T13:57:16.3076965Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3077165Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3077169Z 
2025-12-04T13:57:16.3077255Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3077257Z 
2025-12-04T13:57:16.3077317Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.3077361Z Traceback (most recent call last):
2025-12-04T13:57:16.3077525Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3077566Z     getattr(self, test_name)()
2025-12-04T13:57:16.3077725Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3077759Z     fn()
2025-12-04T13:57:16.3077908Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3077948Z     method(*args, **kwargs)
2025-12-04T13:57:16.3078098Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3078137Z     method(*args, **kwargs)
2025-12-04T13:57:16.3078287Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3078322Z     with policy():
2025-12-04T13:57:16.3078474Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3078514Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3078825Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 2. CUDA driver allocated memory was 2300575744 and is now 2954887168.
2025-12-04T13:57:16.3078827Z 
2025-12-04T13:57:16.3078900Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3079101Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3079103Z 
2025-12-04T13:57:16.3079189Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3079252Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.3079343Z ======================= 1 failed, 3 deselected in 5.92s ========================
2025-12-04T13:57:16.3079381Z Got exit code 1
2025-12-04T13:57:16.3079423Z Retrying single test...
2025-12-04T13:57:16.3079636Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-5d0f3f05350a1d7a.xml
2025-12-04T13:57:16.3079695Z ============================= test session starts ==============================
2025-12-04T13:57:16.3079807Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.3079876Z cachedir: .pytest_cache
2025-12-04T13:57:16.3080035Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.3080084Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.3080127Z configfile: pytest.ini
2025-12-04T13:57:16.3080293Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.3080365Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.3080563Z stepcurrent: skipping 2 already run items. Running only test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda
2025-12-04T13:57:16.3080607Z Running 1 items in this shard
2025-12-04T13:57:16.3080609Z 
2025-12-04T13:57:16.3080891Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda I1204 13:56:33.658000 345851 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 345920
2025-12-04T13:57:16.3081048Z I1204 13:56:33.659000 345851 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 345921
2025-12-04T13:57:16.3081202Z I1204 13:56:33.660000 345851 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 345922
2025-12-04T13:57:16.3081357Z I1204 13:56:33.660000 345851 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 345923
2025-12-04T13:57:16.3081853Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3081919Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3082407Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3082471Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3082957Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3083017Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3083500Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3083579Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3083874Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.3083917Z   return func(*args, **kwargs)
2025-12-04T13:57:16.3084403Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3084487Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3084973Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3085033Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3085515Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3085577Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3086067Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T13:57:16.3086124Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T13:57:16.3086411Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T13:57:16.3086455Z   return func(*args, **kwargs)
2025-12-04T13:57:16.3086599Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3086798Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3087091Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3087247Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3087536Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3087666Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3087973Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3088123Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3088399Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3088548Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3088851Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3088989Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3089268Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3089414Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3089853Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2462056448 and is now 3114270720.
2025-12-04T13:57:16.3089970Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3090169Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3090497Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3090611Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3090826Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3090991Z [rank0]:E1204 13:56:38.464000 345920 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.3091029Z dist init r=0, world=4
2025-12-04T13:57:16.3091167Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3091326Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3091610Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3091766Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3092050Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3092194Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3092470Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3092616Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3092893Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3093062Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3093341Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3093476Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3093754Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3093903Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3094340Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 1. CUDA driver allocated memory was 2317352960 and is now 2971664384.
2025-12-04T13:57:16.3094455Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3094651Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3094977Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3095092Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3095308Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3095476Z [rank1]:E1204 13:56:38.465000 345921 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.3095517Z dist init r=1, world=4
2025-12-04T13:57:16.3095657Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3095817Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3096107Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3096260Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3096569Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3096693Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3097017Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3097194Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3097473Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3097625Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3097902Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3098040Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3098319Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3098469Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3098905Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 2. CUDA driver allocated memory was 2300575744 and is now 2954887168.
2025-12-04T13:57:16.3099020Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3099217Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3099546Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3099663Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3099878Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3100045Z [rank2]:E1204 13:56:38.482000 345922 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.3100089Z dist init r=2, world=4
2025-12-04T13:57:16.3100226Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3100392Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3100713Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3100870Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3101153Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3101279Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3101573Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3101721Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3102000Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3102148Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3102428Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3102565Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3102846Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3102994Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3103431Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 3. CUDA driver allocated memory was 2250244096 and is now 2904555520.
2025-12-04T13:57:16.3103549Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3103742Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3104069Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3104182Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3104398Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3104564Z [rank3]:E1204 13:56:38.558000 345923 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.3104603Z dist init r=3, world=4
2025-12-04T13:57:16.3104642Z FAILED [5.7119s] [100%]
2025-12-04T13:57:16.3104647Z 
2025-12-04T13:57:16.3104704Z =================================== FAILURES ===================================
2025-12-04T13:57:16.3104820Z _________________ TestClipGradNormCUDA.test_no_gradients_cuda __________________
2025-12-04T13:57:16.3104868Z Traceback (most recent call last):
2025-12-04T13:57:16.3105032Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.3105076Z     self._join_processes(fn)
2025-12-04T13:57:16.3105248Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.3105302Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.3105507Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.3105552Z     raise RuntimeError(error)
2025-12-04T13:57:16.3105635Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.3105681Z Traceback (most recent call last):
2025-12-04T13:57:16.3105844Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3105885Z     getattr(self, test_name)()
2025-12-04T13:57:16.3106044Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3106077Z     fn()
2025-12-04T13:57:16.3106231Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3106273Z     method(*args, **kwargs)
2025-12-04T13:57:16.3106426Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3106465Z     method(*args, **kwargs)
2025-12-04T13:57:16.3106617Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3106654Z     with policy():
2025-12-04T13:57:16.3106854Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3106895Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3107206Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2462056448 and is now 3114270720.
2025-12-04T13:57:16.3107208Z 
2025-12-04T13:57:16.3107285Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3107485Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3107487Z 
2025-12-04T13:57:16.3107578Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3107579Z 
2025-12-04T13:57:16.3107637Z Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.3107687Z Traceback (most recent call last):
2025-12-04T13:57:16.3107850Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3107895Z     getattr(self, test_name)()
2025-12-04T13:57:16.3108052Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3108085Z     fn()
2025-12-04T13:57:16.3108235Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3108279Z     method(*args, **kwargs)
2025-12-04T13:57:16.3108430Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3108472Z     method(*args, **kwargs)
2025-12-04T13:57:16.3108622Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3108688Z     with policy():
2025-12-04T13:57:16.3108840Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3108882Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3109194Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 1. CUDA driver allocated memory was 2317352960 and is now 2971664384.
2025-12-04T13:57:16.3109223Z 
2025-12-04T13:57:16.3109299Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3109500Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3109502Z 
2025-12-04T13:57:16.3109590Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3109592Z 
2025-12-04T13:57:16.3109655Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.3109702Z Traceback (most recent call last):
2025-12-04T13:57:16.3109866Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3109908Z     getattr(self, test_name)()
2025-12-04T13:57:16.3110067Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3110103Z     fn()
2025-12-04T13:57:16.3110253Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3110292Z     method(*args, **kwargs)
2025-12-04T13:57:16.3110442Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3110481Z     method(*args, **kwargs)
2025-12-04T13:57:16.3110634Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3110673Z     with policy():
2025-12-04T13:57:16.3110825Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3110869Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3111179Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 2. CUDA driver allocated memory was 2300575744 and is now 2954887168.
2025-12-04T13:57:16.3111183Z 
2025-12-04T13:57:16.3111259Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3111460Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3111462Z 
2025-12-04T13:57:16.3111552Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3111554Z 
2025-12-04T13:57:16.3111555Z 
2025-12-04T13:57:16.3111631Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.3111720Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.3111980Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-5d0f3f05350a1d7a.xml -
2025-12-04T13:57:16.3112045Z =========================== short test summary info ============================
2025-12-04T13:57:16.3112267Z FAILED [5.7119s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.3112314Z Traceback (most recent call last):
2025-12-04T13:57:16.3112500Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3112546Z     getattr(self, test_name)()
2025-12-04T13:57:16.3112707Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3112741Z     fn()
2025-12-04T13:57:16.3112893Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3112954Z     method(*args, **kwargs)
2025-12-04T13:57:16.3113107Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3113146Z     method(*args, **kwargs)
2025-12-04T13:57:16.3113296Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3113332Z     with policy():
2025-12-04T13:57:16.3113489Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3113529Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3113842Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 0. CUDA driver allocated memory was 2462056448 and is now 3114270720.
2025-12-04T13:57:16.3113844Z 
2025-12-04T13:57:16.3113919Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3114117Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3114119Z 
2025-12-04T13:57:16.3114205Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3114207Z 
2025-12-04T13:57:16.3114265Z Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.3114313Z Traceback (most recent call last):
2025-12-04T13:57:16.3114475Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3114521Z     getattr(self, test_name)()
2025-12-04T13:57:16.3114679Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3114713Z     fn()
2025-12-04T13:57:16.3114861Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3114904Z     method(*args, **kwargs)
2025-12-04T13:57:16.3115052Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3115093Z     method(*args, **kwargs)
2025-12-04T13:57:16.3115245Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3115285Z     with policy():
2025-12-04T13:57:16.3115436Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3115479Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3115789Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 1. CUDA driver allocated memory was 2317352960 and is now 2971664384.
2025-12-04T13:57:16.3115795Z 
2025-12-04T13:57:16.3115867Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3116064Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3116066Z 
2025-12-04T13:57:16.3116153Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3116183Z 
2025-12-04T13:57:16.3116242Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.3116288Z Traceback (most recent call last):
2025-12-04T13:57:16.3116452Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3116494Z     getattr(self, test_name)()
2025-12-04T13:57:16.3116654Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3116711Z     fn()
2025-12-04T13:57:16.3116901Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3116941Z     method(*args, **kwargs)
2025-12-04T13:57:16.3117090Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3117129Z     method(*args, **kwargs)
2025-12-04T13:57:16.3117282Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3117318Z     with policy():
2025-12-04T13:57:16.3117471Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3117512Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3117818Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_no_gradients_cuda! Caching allocator allocated memory was 512 and is now reported as 6656 on device 2. CUDA driver allocated memory was 2300575744 and is now 2954887168.
2025-12-04T13:57:16.3117822Z 
2025-12-04T13:57:16.3117896Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3118090Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_no_gradients_cuda
2025-12-04T13:57:16.3118092Z 
2025-12-04T13:57:16.3118182Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3118244Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.3118309Z ======================= 1 failed, 3 deselected in 5.72s ========================
2025-12-04T13:57:16.3118347Z Got exit code 1
2025-12-04T13:57:16.3118504Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda
2025-12-04T13:57:16.3118634Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T13:57:16.3118849Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-b687cf440d034fef.xml
2025-12-04T13:57:16.3118907Z ============================= test session starts ==============================
2025-12-04T13:57:16.3119023Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.3119066Z cachedir: .pytest_cache
2025-12-04T13:57:16.3119226Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.3119272Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.3119312Z configfile: pytest.ini
2025-12-04T13:57:16.3119475Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.3119546Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.3119600Z stepcurrent: skipping 3 already run items.
2025-12-04T13:57:16.3119644Z Running 1 items in this shard
2025-12-04T13:57:16.3119645Z 
2025-12-04T13:57:16.3119951Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda I1204 13:56:41.968000 346229 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 346298
2025-12-04T13:57:16.3120106Z I1204 13:56:41.968000 346229 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 346299
2025-12-04T13:57:16.3120258Z I1204 13:56:41.969000 346229 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 346300
2025-12-04T13:57:16.3120407Z I1204 13:56:41.969000 346229 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 346301
2025-12-04T13:57:16.3120549Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3120737Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3121031Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3121187Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3121473Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3121599Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3121881Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3122031Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3122309Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3122458Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3122737Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3122875Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3123155Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3123302Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3123737Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 2. CUDA driver allocated memory was 2300575744 and is now 3258974208.
2025-12-04T13:57:16.3123854Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3124051Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3124396Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3124510Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3124723Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3124908Z [rank2]:E1204 13:56:48.765000 346300 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.3124947Z dist init r=2, world=4
2025-12-04T13:57:16.3125084Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3125245Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3125529Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3125683Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3125966Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3126092Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3126373Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3126522Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3126844Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3126992Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3127266Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3127404Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3127683Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3127831Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3128259Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 3. CUDA driver allocated memory was 2250244096 and is now 3208642560.
2025-12-04T13:57:16.3128375Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3128594Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3128918Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3129032Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3129274Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3129438Z [rank3]:E1204 13:56:48.809000 346301 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.3129476Z dist init r=3, world=4
2025-12-04T13:57:16.3129616Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3129774Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3130060Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3130215Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3130499Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3130625Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3130901Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3131049Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3131325Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3131472Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3131748Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3131885Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3132164Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3132313Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3132764Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 0. CUDA driver allocated memory was 2462056448 and is now 3418357760.
2025-12-04T13:57:16.3132878Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3133078Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3133404Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3133540Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3133754Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3133919Z [rank0]:E1204 13:56:48.842000 346298 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.3133960Z dist init r=0, world=4
2025-12-04T13:57:16.3134099Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3134260Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3134548Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3134703Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3134988Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3135111Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3135388Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3135538Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3135820Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3135965Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3136243Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3136378Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3136659Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3136840Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3137298Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 2317352960 and is now 3275751424.
2025-12-04T13:57:16.3137414Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3137608Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3137958Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3138071Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3138283Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3138447Z [rank1]:E1204 13:56:48.921000 346299 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.3138484Z dist init r=1, world=4
2025-12-04T13:57:16.3138823Z [rank0]:[W1204 13:56:49.591021068 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.3138861Z FAILED [8.8135s] [100%]
2025-12-04T13:57:16.3138863Z 
2025-12-04T13:57:16.3138919Z =================================== FAILURES ===================================
2025-12-04T13:57:16.3139007Z ___________________ TestClipGradNormCUDA.test_non_root_cuda ____________________
2025-12-04T13:57:16.3139056Z Traceback (most recent call last):
2025-12-04T13:57:16.3139219Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.3139266Z     self._join_processes(fn)
2025-12-04T13:57:16.3139440Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.3139496Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.3139674Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.3139718Z     raise RuntimeError(error)
2025-12-04T13:57:16.3139798Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.3139844Z Traceback (most recent call last):
2025-12-04T13:57:16.3140006Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3140048Z     getattr(self, test_name)()
2025-12-04T13:57:16.3140205Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3140240Z     fn()
2025-12-04T13:57:16.3140389Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3140433Z     method(*args, **kwargs)
2025-12-04T13:57:16.3140582Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3140623Z     method(*args, **kwargs)
2025-12-04T13:57:16.3140773Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3140809Z     with policy():
2025-12-04T13:57:16.3140981Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3141022Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3141326Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 0. CUDA driver allocated memory was 2462056448 and is now 3418357760.
2025-12-04T13:57:16.3141329Z 
2025-12-04T13:57:16.3141427Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3141623Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3141625Z 
2025-12-04T13:57:16.3141713Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3141715Z 
2025-12-04T13:57:16.3141777Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.3141821Z Traceback (most recent call last):
2025-12-04T13:57:16.3141983Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3142024Z     getattr(self, test_name)()
2025-12-04T13:57:16.3142181Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3142217Z     fn()
2025-12-04T13:57:16.3142372Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3142414Z     method(*args, **kwargs)
2025-12-04T13:57:16.3142564Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3142604Z     method(*args, **kwargs)
2025-12-04T13:57:16.3142754Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3142791Z     with policy():
2025-12-04T13:57:16.3142943Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3142983Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3143286Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 2. CUDA driver allocated memory was 2300575744 and is now 3258974208.
2025-12-04T13:57:16.3143290Z 
2025-12-04T13:57:16.3143364Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3143557Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3143559Z 
2025-12-04T13:57:16.3143648Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3143650Z 
2025-12-04T13:57:16.3143652Z 
2025-12-04T13:57:16.3143728Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.3143815Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.3144077Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-b687cf440d034fef.xml -
2025-12-04T13:57:16.3144139Z =========================== short test summary info ============================
2025-12-04T13:57:16.3144352Z FAILED [8.8135s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T13:57:16.3144399Z Traceback (most recent call last):
2025-12-04T13:57:16.3144587Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3144628Z     getattr(self, test_name)()
2025-12-04T13:57:16.3144790Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3144823Z     fn()
2025-12-04T13:57:16.3144973Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3145013Z     method(*args, **kwargs)
2025-12-04T13:57:16.3145162Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3145231Z     method(*args, **kwargs)
2025-12-04T13:57:16.3145382Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3145419Z     with policy():
2025-12-04T13:57:16.3145572Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3145611Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3145916Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 0. CUDA driver allocated memory was 2462056448 and is now 3418357760.
2025-12-04T13:57:16.3145918Z 
2025-12-04T13:57:16.3145992Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3146184Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3146187Z 
2025-12-04T13:57:16.3146274Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3146276Z 
2025-12-04T13:57:16.3146335Z Process 2 exited with error code 10 and exception:
2025-12-04T13:57:16.3146381Z Traceback (most recent call last):
2025-12-04T13:57:16.3146543Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3146586Z     getattr(self, test_name)()
2025-12-04T13:57:16.3146796Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3146831Z     fn()
2025-12-04T13:57:16.3146980Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3147025Z     method(*args, **kwargs)
2025-12-04T13:57:16.3147173Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3147215Z     method(*args, **kwargs)
2025-12-04T13:57:16.3147363Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3147401Z     with policy():
2025-12-04T13:57:16.3147553Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3147594Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3147899Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 2. CUDA driver allocated memory was 2300575744 and is now 3258974208.
2025-12-04T13:57:16.3147901Z 
2025-12-04T13:57:16.3147976Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3148171Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3148174Z 
2025-12-04T13:57:16.3148260Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3148353Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.3148417Z ======================= 1 failed, 3 deselected in 8.82s ========================
2025-12-04T13:57:16.3148455Z Got exit code 1
2025-12-04T13:57:16.3148495Z Retrying single test...
2025-12-04T13:57:16.3148711Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-77aa08f2d8b78d04.xml
2025-12-04T13:57:16.3148768Z ============================= test session starts ==============================
2025-12-04T13:57:16.3148909Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.3148950Z cachedir: .pytest_cache
2025-12-04T13:57:16.3149111Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.3149156Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.3149199Z configfile: pytest.ini
2025-12-04T13:57:16.3149365Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.3149437Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.3149628Z stepcurrent: skipping 3 already run items. Running only test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda
2025-12-04T13:57:16.3149670Z Running 1 items in this shard
2025-12-04T13:57:16.3149672Z 
2025-12-04T13:57:16.3149948Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda I1204 13:56:53.297000 346631 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 346700
2025-12-04T13:57:16.3150104Z I1204 13:56:53.298000 346631 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 346701
2025-12-04T13:57:16.3150256Z I1204 13:56:53.298000 346631 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 346702
2025-12-04T13:57:16.3150407Z I1204 13:56:53.299000 346631 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 346703
2025-12-04T13:57:16.3150550Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3150712Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3151003Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3151162Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3151451Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3151578Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3151859Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3152010Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3152289Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3152460Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3152737Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3152874Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3153174Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3153321Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3153759Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 2317352960 and is now 3275751424.
2025-12-04T13:57:16.3153873Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3154070Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3154393Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3154506Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3154720Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3154884Z [rank1]:E1204 13:57:00.161000 346701 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.3154923Z dist init r=1, world=4
2025-12-04T13:57:16.3155059Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3155220Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3155505Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3155661Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3155949Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3156072Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3156350Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3156496Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3156849Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3156996Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3157271Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3157441Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3157717Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3157867Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3158304Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 3. CUDA driver allocated memory was 2250244096 and is now 3208642560.
2025-12-04T13:57:16.3158420Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3158613Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3158935Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3159047Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3159259Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3159424Z [rank3]:E1204 13:57:00.176000 346703 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.3159464Z dist init r=3, world=4
2025-12-04T13:57:16.3159601Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3159760Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3160050Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3160203Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3160493Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3160619Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3160924Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3161071Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3161345Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3161492Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3161790Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3161926Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3162206Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3162352Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3162786Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 2. CUDA driver allocated memory was 2300575744 and is now 3258974208.
2025-12-04T13:57:16.3162901Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3163098Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3163415Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3163527Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3163742Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3163905Z [rank2]:E1204 13:57:00.192000 346702 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.3163944Z dist init r=2, world=4
2025-12-04T13:57:16.3164083Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3164244Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3164529Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3164686Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3164970Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3165118Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3165395Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3165541Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3165815Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3165981Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3166258Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3166393Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3166673Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3166867Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3167305Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 0. CUDA driver allocated memory was 2459959296 and is now 3418357760.
2025-12-04T13:57:16.3167419Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3167614Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3167936Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3168051Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3168263Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3168428Z [rank0]:E1204 13:57:00.297000 346700 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.3168465Z dist init r=0, world=4
2025-12-04T13:57:16.3168804Z [rank0]:[W1204 13:57:00.044654255 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.3168842Z FAILED [8.8138s] [100%]
2025-12-04T13:57:16.3168846Z 
2025-12-04T13:57:16.3168903Z =================================== FAILURES ===================================
2025-12-04T13:57:16.3168988Z ___________________ TestClipGradNormCUDA.test_non_root_cuda ____________________
2025-12-04T13:57:16.3169034Z Traceback (most recent call last):
2025-12-04T13:57:16.3169196Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.3169268Z     self._join_processes(fn)
2025-12-04T13:57:16.3169442Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.3169497Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.3169675Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.3169719Z     raise RuntimeError(error)
2025-12-04T13:57:16.3169829Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.3169877Z Traceback (most recent call last):
2025-12-04T13:57:16.3170037Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3170080Z     getattr(self, test_name)()
2025-12-04T13:57:16.3170238Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3170274Z     fn()
2025-12-04T13:57:16.3170424Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3170465Z     method(*args, **kwargs)
2025-12-04T13:57:16.3170616Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3170654Z     method(*args, **kwargs)
2025-12-04T13:57:16.3170804Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3170842Z     with policy():
2025-12-04T13:57:16.3170994Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3171034Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3171341Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 2317352960 and is now 3275751424.
2025-12-04T13:57:16.3171343Z 
2025-12-04T13:57:16.3171418Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3171613Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3171616Z 
2025-12-04T13:57:16.3171705Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3171706Z 
2025-12-04T13:57:16.3171708Z 
2025-12-04T13:57:16.3171783Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.3171871Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.3172133Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-77aa08f2d8b78d04.xml -
2025-12-04T13:57:16.3172195Z =========================== short test summary info ============================
2025-12-04T13:57:16.3172407Z FAILED [8.8138s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T13:57:16.3172454Z Traceback (most recent call last):
2025-12-04T13:57:16.3172618Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3172662Z     getattr(self, test_name)()
2025-12-04T13:57:16.3172820Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3172854Z     fn()
2025-12-04T13:57:16.3173004Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3173066Z     method(*args, **kwargs)
2025-12-04T13:57:16.3173217Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3173257Z     method(*args, **kwargs)
2025-12-04T13:57:16.3173406Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3173443Z     with policy():
2025-12-04T13:57:16.3173593Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3173658Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3173963Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 2317352960 and is now 3275751424.
2025-12-04T13:57:16.3173967Z 
2025-12-04T13:57:16.3174044Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3174238Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3174241Z 
2025-12-04T13:57:16.3174327Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3174389Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.3174451Z ======================= 1 failed, 3 deselected in 8.82s ========================
2025-12-04T13:57:16.3174488Z Got exit code 1
2025-12-04T13:57:16.3174527Z Retrying single test...
2025-12-04T13:57:16.3174742Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-50daeec42c243f1d.xml
2025-12-04T13:57:16.3174798Z ============================= test session starts ==============================
2025-12-04T13:57:16.3174912Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.3174952Z cachedir: .pytest_cache
2025-12-04T13:57:16.3175110Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.3175155Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.3175195Z configfile: pytest.ini
2025-12-04T13:57:16.3175356Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.3175430Z collecting ... collected 4 items / 3 deselected / 1 selected
2025-12-04T13:57:16.3175621Z stepcurrent: skipping 3 already run items. Running only test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda
2025-12-04T13:57:16.3175664Z Running 1 items in this shard
2025-12-04T13:57:16.3175666Z 
2025-12-04T13:57:16.3175943Z distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda I1204 13:57:04.763000 347033 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 347102
2025-12-04T13:57:16.3176097Z I1204 13:57:04.764000 347033 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 347103
2025-12-04T13:57:16.3176248Z I1204 13:57:04.765000 347033 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 347104
2025-12-04T13:57:16.3176398Z I1204 13:57:04.765000 347033 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 347105
2025-12-04T13:57:16.3176541Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3176703Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3177064Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3177219Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3177505Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3177654Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3177934Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3178084Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3178360Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3178507Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3178783Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3178922Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3179202Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3179350Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3179780Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 3. CUDA driver allocated memory was 2250244096 and is now 3208642560.
2025-12-04T13:57:16.3179896Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3180095Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3180417Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3180531Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3180743Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3180908Z [rank3]:E1204 13:57:11.527000 347105 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T13:57:16.3180946Z dist init r=3, world=4
2025-12-04T13:57:16.3181106Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3181267Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3181552Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3181706Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3182011Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3182135Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3182417Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3182565Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3182840Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3182988Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3183266Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3183401Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3183681Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3183831Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3184262Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 2317352960 and is now 3275751424.
2025-12-04T13:57:16.3186357Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3186565Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3186926Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3187049Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3187261Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3187470Z [rank1]:E1204 13:57:11.588000 347103 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T13:57:16.3187511Z dist init r=1, world=4
2025-12-04T13:57:16.3187649Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3187809Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3188097Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3188284Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3188569Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3188693Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3188974Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3189123Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3189399Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3189548Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3189825Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3189961Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3190244Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3190395Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3190827Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 2. CUDA driver allocated memory was 2300575744 and is now 3258974208.
2025-12-04T13:57:16.3190943Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3191139Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3191466Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3191578Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3191813Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3191979Z [rank2]:E1204 13:57:11.600000 347104 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T13:57:16.3192017Z dist init r=2, world=4
2025-12-04T13:57:16.3192156Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T13:57:16.3192342Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T13:57:16.3192629Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3192783Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T13:57:16.3193067Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3193190Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T13:57:16.3193469Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3193618Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3193896Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3194044Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T13:57:16.3194319Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3194457Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T13:57:16.3194734Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3194884Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T13:57:16.3195315Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 0. CUDA driver allocated memory was 2459959296 and is now 3418357760.
2025-12-04T13:57:16.3195430Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3195626Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3195970Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3196087Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T13:57:16.3196298Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3196462Z [rank0]:E1204 13:57:11.618000 347102 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T13:57:16.3196522Z dist init r=0, world=4
2025-12-04T13:57:16.3196900Z [rank0]:[W1204 13:57:11.271864867 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T13:57:16.3196941Z FAILED [8.5155s] [100%]
2025-12-04T13:57:16.3196945Z 
2025-12-04T13:57:16.3197003Z =================================== FAILURES ===================================
2025-12-04T13:57:16.3197091Z ___________________ TestClipGradNormCUDA.test_non_root_cuda ____________________
2025-12-04T13:57:16.3197138Z Traceback (most recent call last):
2025-12-04T13:57:16.3197305Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T13:57:16.3197349Z     self._join_processes(fn)
2025-12-04T13:57:16.3197525Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T13:57:16.3197578Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T13:57:16.3197758Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T13:57:16.3197801Z     raise RuntimeError(error)
2025-12-04T13:57:16.3197884Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.3197929Z Traceback (most recent call last):
2025-12-04T13:57:16.3198093Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3198135Z     getattr(self, test_name)()
2025-12-04T13:57:16.3198293Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3198330Z     fn()
2025-12-04T13:57:16.3198482Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3198523Z     method(*args, **kwargs)
2025-12-04T13:57:16.3198674Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3198714Z     method(*args, **kwargs)
2025-12-04T13:57:16.3198865Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3198904Z     with policy():
2025-12-04T13:57:16.3199055Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3199096Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3199400Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 3. CUDA driver allocated memory was 2250244096 and is now 3208642560.
2025-12-04T13:57:16.3199404Z 
2025-12-04T13:57:16.3199482Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3199679Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3199681Z 
2025-12-04T13:57:16.3199801Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3199803Z 
2025-12-04T13:57:16.3199805Z 
2025-12-04T13:57:16.3199882Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T13:57:16.3199970Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T13:57:16.3200235Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-50daeec42c243f1d.xml -
2025-12-04T13:57:16.3200324Z =========================== short test summary info ============================
2025-12-04T13:57:16.3200538Z FAILED [8.5155s] distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T13:57:16.3200583Z Traceback (most recent call last):
2025-12-04T13:57:16.3200749Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T13:57:16.3200792Z     getattr(self, test_name)()
2025-12-04T13:57:16.3200952Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T13:57:16.3200986Z     fn()
2025-12-04T13:57:16.3201137Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3201180Z     method(*args, **kwargs)
2025-12-04T13:57:16.3201337Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T13:57:16.3201377Z     method(*args, **kwargs)
2025-12-04T13:57:16.3201527Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T13:57:16.3201562Z     with policy():
2025-12-04T13:57:16.3201717Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T13:57:16.3201757Z     raise RuntimeError(msg)
2025-12-04T13:57:16.3202066Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestClipGradNormCUDA.test_non_root_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 3. CUDA driver allocated memory was 2250244096 and is now 3208642560.
2025-12-04T13:57:16.3202069Z 
2025-12-04T13:57:16.3202143Z To execute this test, run the following from the base repo dir:
2025-12-04T13:57:16.3202338Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_clip_grad_norm.py TestClipGradNormCUDA.test_non_root_cuda
2025-12-04T13:57:16.3202341Z 
2025-12-04T13:57:16.3202431Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T13:57:16.3202493Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T13:57:16.3202559Z ======================= 1 failed, 3 deselected in 8.52s ========================
2025-12-04T13:57:16.3202597Z Got exit code 1
2025-12-04T13:57:16.3202744Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda
2025-12-04T13:57:16.3202875Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T13:57:16.3203094Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-93eececb0d09ed21.xml
2025-12-04T13:57:16.3203155Z ============================= test session starts ==============================
2025-12-04T13:57:16.3203267Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T13:57:16.3203309Z cachedir: .pytest_cache
2025-12-04T13:57:16.3203472Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T13:57:16.3203548Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T13:57:16.3203595Z configfile: pytest.ini
2025-12-04T13:57:16.3203759Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T13:57:16.3203832Z collecting ... collected 4 items / 4 deselected / 0 selected
2025-12-04T13:57:16.3203886Z stepcurrent: skipping 4 already run items.
2025-12-04T13:57:16.3203931Z Running 0 items in this shard
2025-12-04T13:57:16.3203958Z 
2025-12-04T13:57:16.3204217Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-93eececb0d09ed21.xml -
2025-12-04T13:57:16.3204277Z ============================ 4 deselected in 0.00s =============================
2025-12-04T13:57:16.3204857Z The following tests failed consistently: ['test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda', 'test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda', 'test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda', 'test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda']
2025-12-04T13:57:16.3204860Z 
2025-12-04T13:57:16.3205066Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm 1/1 (test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_5e6bd88c9575523a_.log)
2025-12-04T13:57:16.3205070Z 
2025-12-04T13:57:16.3205204Z Finished distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2025-12-04 13:57:16.253995][2240460.711314017], took 2.43min
2025-12-04T13:57:16.3205485Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:57:16.3205572Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:57:16.3205670Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-12-04T13:57:16.3205718Z Uploading artifacts took 0.00 seconds
2025-12-04T13:57:16.3205788Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 failed!
2025-12-04T13:57:16.3205915Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2025-12-04 13:57:16.256763][2240460.714087972]
2025-12-04T13:57:16.3205965Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:57:16.3206298Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:57:16.256936]
2025-12-04T13:57:18.2517425Z 
2025-12-04T13:57:18.2518206Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_23c2c25eb5643ad8_.log
2025-12-04T13:57:18.2518632Z Running 0 items in this shard:
2025-12-04T13:57:18.2518721Z 
2025-12-04T13:57:18.2518871Z Finished distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2025-12-04 13:57:18.251460][2240462.708780242], took 0.03min
2025-12-04T13:57:18.2531866Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:57:18.2539811Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:57:18.2541089Z Running distributed/nn/jit/test_instantiator 1/1 ... [2025-12-04 13:57:18.254005][2240462.711330132]
2025-12-04T13:57:18.2541299Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:57:18.2543548Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:57:18.254186]
2025-12-04T13:57:20.3215994Z 
2025-12-04T13:57:20.3216694Z distributed/nn/jit/test_instantiator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.nn.jit.test_instantiator_1.1_83d393f54f59e7a3_.log
2025-12-04T13:57:20.3218578Z Running 3 items in this shard: test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_get_arg_return_types_from_interface, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_non_scripted_remote_module_template, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_scripted_remote_module_template
2025-12-04T13:57:20.3220133Z 
2025-12-04T13:57:20.3220451Z Finished distributed/nn/jit/test_instantiator 1/1 ... [2025-12-04 13:57:20.321291][2240464.778611692], took 0.03min
2025-12-04T13:57:20.3235734Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:57:20.3242073Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:57:20.3245241Z Running distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2025-12-04 13:57:20.324312][2240464.781636764]
2025-12-04T13:57:20.3245502Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:57:20.3246168Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:57:20.324483]
2025-12-04T13:57:34.4113493Z 
2025-12-04T13:57:34.4114513Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_76c69056f55da171_.log
2025-12-04T13:57:34.4119149Z Running 11 items in this shard: test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_check_overlapping, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_chunked_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_device_placement, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_enumerable_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunk_sharding_params, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunked_dim_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_split_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_infer_sharding_spec_from_shards_metadata, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_shard_tensor, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_tensor_ctor
2025-12-04T13:57:34.4122809Z 
2025-12-04T13:57:34.4123118Z Finished distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2025-12-04 13:57:34.411062][2240478.868380583], took 0.23min
2025-12-04T13:57:34.4136810Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:57:34.4143757Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:57:34.4147138Z Running distributed/fsdp/test_fsdp_misc 1/1 ... [2025-12-04 13:57:34.414474][2240478.871798049]
2025-12-04T13:57:34.4147698Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:57:34.4149443Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:57:34.414687]
2025-12-04T13:58:51.9032794Z 
2025-12-04T13:58:51.9033927Z distributed/fsdp/test_fsdp_misc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_c2e6a4ccafd6f874_.log
2025-12-04T13:58:51.9043931Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_cpu_init_with_sync_module_states, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_init_stays_on_cpu, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_training, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_False, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_True, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_not_all_outputs_used_in_loss, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optim_overlap_no_use_orig_params_error, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optimizer_overlap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_zero2_eval_with_prefetch, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_cpu_gpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_device_id_auto_wrap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_cpu_offload, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_no_move_ignored_params_and_bufs, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_ignored_module_meta, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_namedtuple, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_same_model_across_ranks, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_unsupported_module_cls, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_homogeneous_attributes, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_module_device_mismatches_device_id, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_multigpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_no_params, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_training_device_mismatch_errors, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_unsafe_setattr, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_world_size_1_sharding_strategy_warning
2025-12-04T13:58:51.9048195Z 
2025-12-04T13:58:51.9048324Z Finished distributed/fsdp/test_fsdp_misc 1/1 ... [2025-12-04 13:58:51.902924][2240556.36024353], took 1.29min
2025-12-04T13:58:51.9054197Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:58:51.9060628Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:58:51.9063157Z Running distributed/_composable/fsdp/test_fully_shard_memory 1/1 ... [2025-12-04 13:58:51.906222][2240556.36354686]
2025-12-04T13:58:51.9063393Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:58:51.9064994Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:58:51.906396]
2025-12-04T13:59:09.1800767Z 
2025-12-04T13:59:09.1801559Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_memory_1.1_e65ad99eea6a8867_.log
2025-12-04T13:59:09.1802758Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_memory.py::TestFullyShardMemory::test_fully_shard_del_memory, test/distributed/_composable/fsdp/test_fully_shard_memory.py::TestFullyShardMemory::test_fully_shard_training_memory
2025-12-04T13:59:09.1803193Z 
2025-12-04T13:59:09.1803356Z Finished distributed/_composable/fsdp/test_fully_shard_memory 1/1 ... [2025-12-04 13:59:09.179839][2240573.637160859], took 0.29min
2025-12-04T13:59:09.1816572Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T13:59:09.1823142Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T13:59:09.1825588Z Running distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2025-12-04 13:59:09.182485][2240573.639810071]
2025-12-04T13:59:09.1825905Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T13:59:09.1827511Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 13:59:09.182662]
2025-12-04T14:03:34.9942137Z 
2025-12-04T14:03:34.9943491Z distributed/optim/test_zero_redundancy_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_e95c235e77c56ab2_.log
2025-12-04T14:03:34.9964030Z Running 42 items in this shard: test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_constructor, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_same_dense_param_type, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_state_dict, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_extra_inner_key, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_kwargs, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_without_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_zero_grad, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_add_param_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_collect_shards, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_multiple_param_groups, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_nondefault_process_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_sharding, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step_with_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_cpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_gpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_True
2025-12-04T14:03:34.9976313Z 
2025-12-04T14:03:34.9976485Z Finished distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2025-12-04 14:03:34.994252][2240839.451572541], took 4.43min
2025-12-04T14:03:34.9977042Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:03:34.9977470Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:03:34.9977726Z Running distributed/fsdp/test_fsdp_core 2/3 ... [2025-12-04 14:03:34.997457][2240839.454781245]
2025-12-04T14:03:34.9977938Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:03:34.9978373Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=2', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:03:34.997627]
2025-12-04T14:26:42.1175285Z 
2025-12-04T14:26:42.1176113Z PRINTING LOG FILE of distributed/fsdp/test_fsdp_core 2/3 (test/test-reports/distributed.fsdp.test_fsdp_core_2.3_d5fedc741f801f96_.log)
2025-12-04T14:26:42.1177013Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-5a58f00d223524b2.xml
2025-12-04T14:26:42.1177609Z ============================= test session starts ==============================
2025-12-04T14:26:42.1178029Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1178410Z cachedir: .pytest_cache
2025-12-04T14:26:42.1178906Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1179371Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1179596Z configfile: pytest.ini
2025-12-04T14:26:42.1180030Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1180572Z collecting ... collected 60 items
2025-12-04T14:26:42.1180838Z stepcurrent: Cannot find last run test, not skipping
2025-12-04T14:26:42.1200942Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.1206042Z 
2025-12-04T14:26:42.1206511Z distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda I1204 14:03:36.668000 372296 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 372365
2025-12-04T14:26:42.1207275Z I1204 14:03:36.668000 372296 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 372366
2025-12-04T14:26:42.1207777Z I1204 14:03:36.669000 372296 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 372367
2025-12-04T14:26:42.1208194Z I1204 14:03:36.670000 372296 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 372368
2025-12-04T14:26:42.1209350Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1209918Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1210649Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1211370Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1211944Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1212497Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1213266Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1214003Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1214562Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1215174Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1215898Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1216685Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1217532Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1217980Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1218552Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1219137Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1219392Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1219755Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1220265Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1220760Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1221248Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1221706Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1222158Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1222646Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1223121Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1223635Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1224121Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1224602Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1225109Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1225592Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1226282Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 2300575744 and is now 3380609024.
2025-12-04T14:26:42.1226957Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1227312Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1227945Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1228516Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1228924Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1229390Z [rank2]:E1204 14:03:43.901000 372367 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1229644Z dist init r=2, world=4
2025-12-04T14:26:42.1229855Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1230201Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1230695Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1231181Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1231661Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1232112Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1232558Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1233029Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1233550Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1234018Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1234517Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1235051Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1235569Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1236041Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1236717Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 2250244096 and is now 3330277376.
2025-12-04T14:26:42.1237426Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1237859Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1238512Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1239088Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1239513Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1239977Z [rank3]:E1204 14:03:43.908000 372368 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1240249Z dist init r=3, world=4
2025-12-04T14:26:42.1240467Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1240826Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1241328Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1241848Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1242341Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1242799Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1243299Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1243846Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1244317Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1244815Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1245281Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1245743Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1246203Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1246699Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1247414Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 2317352960 and is now 3397386240.
2025-12-04T14:26:42.1248043Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1248396Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1248994Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1249571Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1249985Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1250407Z [rank1]:E1204 14:03:43.942000 372366 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1250654Z dist init r=1, world=4
2025-12-04T14:26:42.1250859Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1251200Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1251694Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1252176Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1252701Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1253153Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1253602Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1254074Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1254579Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1255046Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1255514Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1255967Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1256433Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1256943Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1257614Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 2453667840 and is now 3533701120.
2025-12-04T14:26:42.1258248Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1258604Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1259199Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1259706Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1260078Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1260494Z [rank0]:E1204 14:03:43.945000 372365 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1260738Z dist init r=0, world=4
2025-12-04T14:26:42.1261167Z [rank0]:[W1204 14:03:44.680222469 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1261584Z FAILED [9.1151s] [  5%]
2025-12-04T14:26:42.1261649Z 
2025-12-04T14:26:42.1261712Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1261909Z ___ TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda ____
2025-12-04T14:26:42.1262133Z Traceback (most recent call last):
2025-12-04T14:26:42.1262388Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1262640Z     self._join_processes(fn)
2025-12-04T14:26:42.1262889Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1263162Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1263470Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1263737Z     raise RuntimeError(error)
2025-12-04T14:26:42.1263889Z RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1264066Z Traceback (most recent call last):
2025-12-04T14:26:42.1264314Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1264556Z     getattr(self, test_name)()
2025-12-04T14:26:42.1264787Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1265022Z     fn()
2025-12-04T14:26:42.1265226Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1265475Z     method(*args, **kwargs)
2025-12-04T14:26:42.1265710Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1265944Z     method(*args, **kwargs)
2025-12-04T14:26:42.1266166Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1266398Z     with policy():
2025-12-04T14:26:42.1266614Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1266911Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1267334Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 2300575744 and is now 3380609024.
2025-12-04T14:26:42.1267716Z 
2025-12-04T14:26:42.1267796Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1268140Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1268409Z 
2025-12-04T14:26:42.1268498Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1268630Z 
2025-12-04T14:26:42.1268633Z 
2025-12-04T14:26:42.1268715Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1268922Z Process 2 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1269283Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-5a58f00d223524b2.xml -
2025-12-04T14:26:42.1269614Z =========================== short test summary info ============================
2025-12-04T14:26:42.1269962Z FAILED [9.1151s] distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda - RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1270290Z Traceback (most recent call last):
2025-12-04T14:26:42.1270538Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1270784Z     getattr(self, test_name)()
2025-12-04T14:26:42.1271070Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1271308Z     fn()
2025-12-04T14:26:42.1271513Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1271744Z     method(*args, **kwargs)
2025-12-04T14:26:42.1271965Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1272195Z     method(*args, **kwargs)
2025-12-04T14:26:42.1272463Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1272698Z     with policy():
2025-12-04T14:26:42.1272909Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1273141Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1273567Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 2300575744 and is now 3380609024.
2025-12-04T14:26:42.1273957Z 
2025-12-04T14:26:42.1274032Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1274374Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1274643Z 
2025-12-04T14:26:42.1274735Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1274928Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1275093Z ============================== 1 failed in 9.27s ===============================
2025-12-04T14:26:42.1275231Z Got exit code 1
2025-12-04T14:26:42.1275333Z Retrying single test...
2025-12-04T14:26:42.1275595Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-f447c467cbe01df3.xml
2025-12-04T14:26:42.1275880Z ============================= test session starts ==============================
2025-12-04T14:26:42.1276095Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1276289Z cachedir: .pytest_cache
2025-12-04T14:26:42.1276515Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1276856Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1276983Z configfile: pytest.ini
2025-12-04T14:26:42.1277214Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1277487Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1277821Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1278122Z Running 1 items in this shard
2025-12-04T14:26:42.1278197Z 
2025-12-04T14:26:42.1278506Z distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda I1204 14:03:48.174000 372698 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 372767
2025-12-04T14:26:42.1279002Z I1204 14:03:48.175000 372698 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 372768
2025-12-04T14:26:42.1279346Z I1204 14:03:48.176000 372698 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 372769
2025-12-04T14:26:42.1279690Z I1204 14:03:48.176000 372698 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 372770
2025-12-04T14:26:42.1280295Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1280740Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1281318Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1281943Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1282402Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1282846Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1283419Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1284011Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1284462Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1284902Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1285475Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1286061Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1286514Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1287000Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1287573Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1288161Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1288405Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1288753Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1289249Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1289766Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1290256Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1290709Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1291187Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1291657Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1292124Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1292589Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1293058Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1293516Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1293973Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1294441Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1295114Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 2453667840 and is now 3533701120.
2025-12-04T14:26:42.1295748Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1296100Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1296697Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1297263Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1297633Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1298055Z [rank0]:E1204 14:03:55.393000 372767 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1298301Z dist init r=0, world=4
2025-12-04T14:26:42.1298507Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1298883Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1299378Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1299862Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1300342Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1300833Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1301279Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1301744Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1302210Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1302684Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1303148Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1303608Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1304068Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1304542Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1305212Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 2300575744 and is now 3380609024.
2025-12-04T14:26:42.1305838Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1306189Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1307031Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1307541Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1307911Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1308328Z [rank2]:E1204 14:03:55.413000 372769 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1308607Z dist init r=2, world=4
2025-12-04T14:26:42.1308816Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1309155Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1309645Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1310166Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1310651Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1311105Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1311548Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1312015Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1312486Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1312952Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1313423Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1313879Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1314338Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1314808Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1315474Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 2250244096 and is now 3330277376.
2025-12-04T14:26:42.1316099Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1316452Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1317093Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1317608Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1318005Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1318431Z [rank3]:E1204 14:03:55.435000 372770 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1318678Z dist init r=3, world=4
2025-12-04T14:26:42.1318884Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1319256Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1319747Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1320237Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1320727Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1321186Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1321637Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1322112Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1322579Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1323048Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1323514Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1323988Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1324457Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1324935Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1325606Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 2317352960 and is now 3397386240.
2025-12-04T14:26:42.1326234Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1326596Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1327356Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1327873Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1328257Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1328682Z [rank1]:E1204 14:03:55.439000 372768 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1328972Z dist init r=1, world=4
2025-12-04T14:26:42.1329386Z [rank0]:[W1204 14:03:55.067171372 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1329808Z FAILED [9.0163s] [100%]
2025-12-04T14:26:42.1329886Z 
2025-12-04T14:26:42.1329949Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1330156Z ___ TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda ____
2025-12-04T14:26:42.1330347Z Traceback (most recent call last):
2025-12-04T14:26:42.1330604Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1330862Z     self._join_processes(fn)
2025-12-04T14:26:42.1331125Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1331401Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1331680Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1331951Z     raise RuntimeError(error)
2025-12-04T14:26:42.1332115Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1332282Z Traceback (most recent call last):
2025-12-04T14:26:42.1332524Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1332767Z     getattr(self, test_name)()
2025-12-04T14:26:42.1333002Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1333243Z     fn()
2025-12-04T14:26:42.1333445Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1333678Z     method(*args, **kwargs)
2025-12-04T14:26:42.1333901Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1334131Z     method(*args, **kwargs)
2025-12-04T14:26:42.1334351Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1334579Z     with policy():
2025-12-04T14:26:42.1334792Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1335024Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1335442Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 2453667840 and is now 3533701120.
2025-12-04T14:26:42.1335829Z 
2025-12-04T14:26:42.1335906Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1336248Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1336543Z 
2025-12-04T14:26:42.1336635Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1336800Z 
2025-12-04T14:26:42.1336802Z 
2025-12-04T14:26:42.1336882Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1337085Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1337445Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-f447c467cbe01df3.xml -
2025-12-04T14:26:42.1337823Z =========================== short test summary info ============================
2025-12-04T14:26:42.1338163Z FAILED [9.0163s] distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1338486Z Traceback (most recent call last):
2025-12-04T14:26:42.1338736Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1338979Z     getattr(self, test_name)()
2025-12-04T14:26:42.1339214Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1339447Z     fn()
2025-12-04T14:26:42.1339654Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1339887Z     method(*args, **kwargs)
2025-12-04T14:26:42.1340108Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1340337Z     method(*args, **kwargs)
2025-12-04T14:26:42.1340554Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1340778Z     with policy():
2025-12-04T14:26:42.1340991Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1341222Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1341637Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 2453667840 and is now 3533701120.
2025-12-04T14:26:42.1342021Z 
2025-12-04T14:26:42.1342095Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1342433Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1342720Z 
2025-12-04T14:26:42.1342916Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1343158Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1343369Z ======================= 1 failed, 19 deselected in 9.18s =======================
2025-12-04T14:26:42.1343565Z Got exit code 1
2025-12-04T14:26:42.1343704Z Retrying single test...
2025-12-04T14:26:42.1343998Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-87ffe249422c8b09.xml
2025-12-04T14:26:42.1344337Z ============================= test session starts ==============================
2025-12-04T14:26:42.1344595Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1344828Z cachedir: .pytest_cache
2025-12-04T14:26:42.1345102Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1345388Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1345678Z configfile: pytest.ini
2025-12-04T14:26:42.1345977Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1346277Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1346670Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1347048Z Running 1 items in this shard
2025-12-04T14:26:42.1347181Z 
2025-12-04T14:26:42.1347512Z distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda I1204 14:03:59.765000 373100 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 373169
2025-12-04T14:26:42.1348056Z I1204 14:03:59.765000 373100 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 373170
2025-12-04T14:26:42.1348448Z I1204 14:03:59.766000 373100 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 373171
2025-12-04T14:26:42.1348843Z I1204 14:03:59.767000 373100 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 373172
2025-12-04T14:26:42.1349676Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1350174Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1350800Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1351426Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1351932Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1352405Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1352871Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1353358Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1353822Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1354306Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1354925Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1355564Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1356233Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1356895Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1357517Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1358181Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1358461Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1358852Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1359392Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1359902Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1360940Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1361451Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1361963Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1362461Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1362966Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1363487Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1363991Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1364486Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1364982Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1365485Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1366198Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 2300575744 and is now 3380609024.
2025-12-04T14:26:42.1366915Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1367347Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1367978Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1368531Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1368971Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1369425Z [rank2]:E1204 14:04:06.995000 373171 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1369697Z dist init r=2, world=4
2025-12-04T14:26:42.1369956Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1370336Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1370862Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1371393Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1371903Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1372402Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1372888Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1373412Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1373922Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1374420Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1374932Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1375416Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1375903Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1376441Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1377254Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 2250244096 and is now 3330277376.
2025-12-04T14:26:42.1377921Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1378313Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1378938Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1379538Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1379947Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1380403Z [rank3]:E1204 14:04:07.000000 373172 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1380684Z dist init r=3, world=4
2025-12-04T14:26:42.1381835Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1382222Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1382754Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1383269Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1383800Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1384285Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1384767Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1385280Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1385781Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1386294Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1386860Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1387350Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1387852Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1388382Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1389096Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 2453667840 and is now 3533701120.
2025-12-04T14:26:42.1389755Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1390168Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1390818Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1391357Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1391776Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1392238Z [rank0]:E1204 14:04:07.012000 373169 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1392528Z dist init r=0, world=4
2025-12-04T14:26:42.1392791Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1393167Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1393703Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1394222Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1394741Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1395235Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1395712Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1396210Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1396730Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1397299Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1397804Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1398297Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1398827Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1399353Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1400053Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 2317352960 and is now 3397386240.
2025-12-04T14:26:42.1400748Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1401147Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1401775Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1402327Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1402734Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1403176Z [rank1]:E1204 14:04:07.023000 373170 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1403479Z dist init r=1, world=4
2025-12-04T14:26:42.1403917Z [rank0]:[W1204 14:04:07.737021812 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1404376Z FAILED [9.1159s] [100%]
2025-12-04T14:26:42.1404472Z 
2025-12-04T14:26:42.1404540Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1404768Z ___ TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda ____
2025-12-04T14:26:42.1405022Z Traceback (most recent call last):
2025-12-04T14:26:42.1405305Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1405610Z     self._join_processes(fn)
2025-12-04T14:26:42.1405898Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1406206Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1406526Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1406864Z     raise RuntimeError(error)
2025-12-04T14:26:42.1407052Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1407266Z Traceback (most recent call last):
2025-12-04T14:26:42.1407547Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1407836Z     getattr(self, test_name)()
2025-12-04T14:26:42.1408106Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1408376Z     fn()
2025-12-04T14:26:42.1408631Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1408935Z     method(*args, **kwargs)
2025-12-04T14:26:42.1409190Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1409478Z     method(*args, **kwargs)
2025-12-04T14:26:42.1409735Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1410002Z     with policy():
2025-12-04T14:26:42.1410257Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1410536Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1435827Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 2250244096 and is now 3330277376.
2025-12-04T14:26:42.1436230Z 
2025-12-04T14:26:42.1436319Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1436670Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1436983Z 
2025-12-04T14:26:42.1437079Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1437205Z 
2025-12-04T14:26:42.1437207Z 
2025-12-04T14:26:42.1437291Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1437501Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1437864Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-87ffe249422c8b09.xml -
2025-12-04T14:26:42.1438197Z =========================== short test summary info ============================
2025-12-04T14:26:42.1438550Z FAILED [9.1159s] distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1438879Z Traceback (most recent call last):
2025-12-04T14:26:42.1439129Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1439373Z     getattr(self, test_name)()
2025-12-04T14:26:42.1439611Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1439848Z     fn()
2025-12-04T14:26:42.1440055Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1440286Z     method(*args, **kwargs)
2025-12-04T14:26:42.1440512Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1440746Z     method(*args, **kwargs)
2025-12-04T14:26:42.1440966Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1441195Z     with policy():
2025-12-04T14:26:42.1441409Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1441645Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1442069Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 2250244096 and is now 3330277376.
2025-12-04T14:26:42.1442461Z 
2025-12-04T14:26:42.1442537Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1442950Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1443218Z 
2025-12-04T14:26:42.1443308Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1443497Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1443661Z ======================= 1 failed, 19 deselected in 9.28s =======================
2025-12-04T14:26:42.1443806Z Got exit code 1
2025-12-04T14:26:42.1444086Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda
2025-12-04T14:26:42.1444433Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.1444793Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-61acadf90bb9d635.xml
2025-12-04T14:26:42.1445088Z ============================= test session starts ==============================
2025-12-04T14:26:42.1445300Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1445485Z cachedir: .pytest_cache
2025-12-04T14:26:42.1445715Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1445955Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1446076Z configfile: pytest.ini
2025-12-04T14:26:42.1446309Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1446583Z collecting ... collected 60 items / 1 deselected / 59 selected
2025-12-04T14:26:42.1446792Z stepcurrent: skipping 1 already run items.
2025-12-04T14:26:42.1446924Z Running 19 items in this shard
2025-12-04T14:26:42.1447000Z 
2025-12-04T14:26:42.1447339Z distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda I1204 14:04:11.216000 373502 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 373571
2025-12-04T14:26:42.1447869Z I1204 14:04:11.217000 373502 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 373572
2025-12-04T14:26:42.1448218Z I1204 14:04:11.218000 373502 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 373573
2025-12-04T14:26:42.1448566Z I1204 14:04:11.218000 373502 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 373574
2025-12-04T14:26:42.1449128Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1449573Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1450013Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1450448Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1451031Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1451633Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1452257Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1452843Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1453294Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1453785Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1454361Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1454943Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1455397Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1455837Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1456411Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1457039Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1457280Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1457625Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1458119Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1458605Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1459124Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1459573Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1460019Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1460485Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1460953Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1461416Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1461913Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1462369Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1462832Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1463334Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1464035Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3026190336.
2025-12-04T14:26:42.1464685Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1465036Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1465658Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1466192Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1466564Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1467024Z [rank3]:E1204 14:04:17.308000 373574 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1467272Z dist init r=3, world=4
2025-12-04T14:26:42.1467486Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1467826Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1468319Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1468803Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1469289Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1469741Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1470188Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1470652Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1471147Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1471616Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1472084Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1472566Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1473020Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1473488Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1474173Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 2. CUDA driver allocated memory was 2300575744 and is now 3076521984.
2025-12-04T14:26:42.1474819Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1475174Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1475793Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1476320Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1476693Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1477165Z [rank2]:E1204 14:04:17.350000 373573 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1477408Z dist init r=2, world=4
2025-12-04T14:26:42.1477613Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1477954Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1478443Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1478921Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1479400Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1479852Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1480332Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1482939Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1483406Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1483895Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1484365Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1484831Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1485290Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1485783Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1486474Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3229614080.
2025-12-04T14:26:42.1487167Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1487521Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1488136Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1488672Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1489039Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1489459Z [rank0]:E1204 14:04:17.368000 373571 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1489706Z dist init r=0, world=4
2025-12-04T14:26:42.1489911Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1490252Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1490741Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1491228Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1491744Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1492205Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1492720Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1493190Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1493680Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1494150Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1494618Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1495074Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1495531Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1495998Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1496690Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3093299200.
2025-12-04T14:26:42.1497383Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1497732Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1498346Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1498877Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1499244Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1499662Z [rank1]:E1204 14:04:17.377000 373572 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1499902Z dist init r=1, world=4
2025-12-04T14:26:42.1500306Z [rank0]:[W1204 14:04:17.239282607 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1500717Z FAILED [7.8152s] [  5%]
2025-12-04T14:26:42.1500781Z 
2025-12-04T14:26:42.1500844Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1501089Z _ TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda _
2025-12-04T14:26:42.1501296Z Traceback (most recent call last):
2025-12-04T14:26:42.1501545Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1501822Z     self._join_processes(fn)
2025-12-04T14:26:42.1502075Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1502358Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1502625Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1502883Z     raise RuntimeError(error)
2025-12-04T14:26:42.1503034Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1503194Z Traceback (most recent call last):
2025-12-04T14:26:42.1503434Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1503674Z     getattr(self, test_name)()
2025-12-04T14:26:42.1503903Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1504137Z     fn()
2025-12-04T14:26:42.1504337Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1504567Z     method(*args, **kwargs)
2025-12-04T14:26:42.1504786Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1505015Z     method(*args, **kwargs)
2025-12-04T14:26:42.1505231Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1505457Z     with policy():
2025-12-04T14:26:42.1505668Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1505897Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1506337Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3026190336.
2025-12-04T14:26:42.1506777Z 
2025-12-04T14:26:42.1506853Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1507214Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1507499Z 
2025-12-04T14:26:42.1507589Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1507713Z 
2025-12-04T14:26:42.1507716Z 
2025-12-04T14:26:42.1507796Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1507997Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1508355Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-61acadf90bb9d635.xml -
2025-12-04T14:26:42.1508684Z =========================== short test summary info ============================
2025-12-04T14:26:42.1509048Z FAILED [7.8152s] distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1509393Z Traceback (most recent call last):
2025-12-04T14:26:42.1509636Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1509914Z     getattr(self, test_name)()
2025-12-04T14:26:42.1510146Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1510396Z     fn()
2025-12-04T14:26:42.1510596Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1510822Z     method(*args, **kwargs)
2025-12-04T14:26:42.1511038Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1511290Z     method(*args, **kwargs)
2025-12-04T14:26:42.1511506Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1511729Z     with policy():
2025-12-04T14:26:42.1511937Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1512171Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1512609Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3026190336.
2025-12-04T14:26:42.1513016Z 
2025-12-04T14:26:42.1513090Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1513452Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1513740Z 
2025-12-04T14:26:42.1513827Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1514014Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1514181Z ======================= 1 failed, 1 deselected in 7.98s ========================
2025-12-04T14:26:42.1514316Z Got exit code 1
2025-12-04T14:26:42.1514412Z Retrying single test...
2025-12-04T14:26:42.1514668Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1a4147cea16eec9b.xml
2025-12-04T14:26:42.1514950Z ============================= test session starts ==============================
2025-12-04T14:26:42.1515159Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1515346Z cachedir: .pytest_cache
2025-12-04T14:26:42.1515570Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1515807Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1515922Z configfile: pytest.ini
2025-12-04T14:26:42.1516150Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1516420Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1516818Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1517138Z Running 1 items in this shard
2025-12-04T14:26:42.1517213Z 
2025-12-04T14:26:42.1517541Z distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda I1204 14:04:21.634000 373888 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 373957
2025-12-04T14:26:42.1518061Z I1204 14:04:21.635000 373888 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 373958
2025-12-04T14:26:42.1518403Z I1204 14:04:21.636000 373888 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 373959
2025-12-04T14:26:42.1518783Z I1204 14:04:21.636000 373888 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 373960
2025-12-04T14:26:42.1519355Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1519792Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1520388Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1520976Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1521427Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1521861Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1522426Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1523006Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1523457Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1523889Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1524455Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1525032Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1525480Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1525914Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1526475Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1527108Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1527345Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1527687Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1528209Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1528703Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1529181Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1529645Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1530086Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1530552Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1531016Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1531476Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1531939Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1532387Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1532845Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1533310Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1533996Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3026190336.
2025-12-04T14:26:42.1534643Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1534999Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1535609Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1536136Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1536500Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1536947Z [rank3]:E1204 14:04:27.604000 373960 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1537321Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1537656Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1538157Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1538633Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1539124Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1539569Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1540012Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1540474Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1540934Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1541393Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1541853Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1542304Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1542758Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1543220Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1543908Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3229614080.
2025-12-04T14:26:42.1544550Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1544901Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1545514Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1546038Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1546426Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1546879Z [rank0]:E1204 14:04:27.604000 373957 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1547147Z dist init r=3, world=4
2025-12-04T14:26:42.1547248Z dist init r=0, world=4
2025-12-04T14:26:42.1547448Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1547782Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1548280Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1548758Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1549240Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1549685Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1550122Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1550583Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1551043Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1551509Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1551971Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1552418Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1552871Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1553335Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1554018Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3093299200.
2025-12-04T14:26:42.1554659Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1555006Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1555646Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1556172Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1556548Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1556997Z [rank1]:E1204 14:04:27.607000 373958 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1557251Z dist init r=1, world=4
2025-12-04T14:26:42.1557450Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1557785Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1558275Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1558751Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1559225Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1559675Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1560113Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1560573Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1561033Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1561492Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1561952Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1562407Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1562859Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1563322Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1564007Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 2. CUDA driver allocated memory was 2300575744 and is now 3076521984.
2025-12-04T14:26:42.1564653Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1565029Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1565636Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1566173Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1566547Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1566998Z [rank2]:E1204 14:04:27.621000 373959 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1567237Z dist init r=2, world=4
2025-12-04T14:26:42.1567636Z [rank0]:[W1204 14:04:27.260178784 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1568044Z FAILED [7.6142s] [100%]
2025-12-04T14:26:42.1568107Z 
2025-12-04T14:26:42.1568163Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1568373Z _ TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda _
2025-12-04T14:26:42.1568573Z Traceback (most recent call last):
2025-12-04T14:26:42.1568815Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1569055Z     self._join_processes(fn)
2025-12-04T14:26:42.1569302Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1569567Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1569839Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1570102Z     raise RuntimeError(error)
2025-12-04T14:26:42.1570258Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1570421Z Traceback (most recent call last):
2025-12-04T14:26:42.1570660Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1570902Z     getattr(self, test_name)()
2025-12-04T14:26:42.1571134Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1571368Z     fn()
2025-12-04T14:26:42.1571572Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1571803Z     method(*args, **kwargs)
2025-12-04T14:26:42.1572024Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1572255Z     method(*args, **kwargs)
2025-12-04T14:26:42.1572475Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1572701Z     with policy():
2025-12-04T14:26:42.1572913Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1573148Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1573591Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3026190336.
2025-12-04T14:26:42.1574039Z 
2025-12-04T14:26:42.1574116Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1574493Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1574785Z 
2025-12-04T14:26:42.1574876Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1575022Z 
2025-12-04T14:26:42.1575024Z 
2025-12-04T14:26:42.1575104Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1575306Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1575665Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1a4147cea16eec9b.xml -
2025-12-04T14:26:42.1576000Z =========================== short test summary info ============================
2025-12-04T14:26:42.1576366Z FAILED [7.6142s] distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1576718Z Traceback (most recent call last):
2025-12-04T14:26:42.1577294Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1577543Z     getattr(self, test_name)()
2025-12-04T14:26:42.1577776Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1578011Z     fn()
2025-12-04T14:26:42.1578214Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1578444Z     method(*args, **kwargs)
2025-12-04T14:26:42.1578665Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1578893Z     method(*args, **kwargs)
2025-12-04T14:26:42.1579113Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1579341Z     with policy():
2025-12-04T14:26:42.1579554Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1579788Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1580235Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3026190336.
2025-12-04T14:26:42.1580639Z 
2025-12-04T14:26:42.1580719Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1581082Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1581368Z 
2025-12-04T14:26:42.1581459Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1581652Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1581820Z ======================= 1 failed, 19 deselected in 7.77s =======================
2025-12-04T14:26:42.1581961Z Got exit code 1
2025-12-04T14:26:42.1582061Z Retrying single test...
2025-12-04T14:26:42.1582318Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-29ae8f388b4c215a.xml
2025-12-04T14:26:42.1582604Z ============================= test session starts ==============================
2025-12-04T14:26:42.1582866Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1583058Z cachedir: .pytest_cache
2025-12-04T14:26:42.1583301Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1583543Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1583663Z configfile: pytest.ini
2025-12-04T14:26:42.1583889Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1584177Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1584534Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1584855Z Running 1 items in this shard
2025-12-04T14:26:42.1584930Z 
2025-12-04T14:26:42.1585266Z distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda I1204 14:04:31.549000 374274 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 374343
2025-12-04T14:26:42.1585787Z I1204 14:04:31.550000 374274 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 374344
2025-12-04T14:26:42.1586132Z I1204 14:04:31.551000 374274 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 374345
2025-12-04T14:26:42.1586480Z I1204 14:04:31.551000 374274 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 374346
2025-12-04T14:26:42.1587081Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1587526Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1588107Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1588699Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1589152Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1589592Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1590170Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1590754Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1591203Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1591639Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1592240Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1592837Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1593287Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.1593738Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.1594307Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.1594893Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.1595135Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1595480Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1595975Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1596460Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1596982Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1597433Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1597879Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1598348Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1598815Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1599282Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1599748Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1600207Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1600671Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1601142Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1603607Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3229614080.
2025-12-04T14:26:42.1604275Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1604625Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1605256Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1605791Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1606157Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1606576Z [rank0]:E1204 14:04:37.493000 374343 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1606868Z dist init r=0, world=4
2025-12-04T14:26:42.1607073Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1607415Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1607907Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1608387Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1608865Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1609313Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1609758Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1610223Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1610687Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1611153Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1611613Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1612066Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1612555Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1613022Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1613722Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3093299200.
2025-12-04T14:26:42.1614382Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1614729Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1615342Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1615875Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1616241Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1616656Z [rank1]:E1204 14:04:37.516000 374344 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1616929Z dist init r=1, world=4
2025-12-04T14:26:42.1617134Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1617475Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1617961Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1618441Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1618924Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1619372Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1619817Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1620282Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1620748Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1621214Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1621678Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1622170Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1622647Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1623110Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1623808Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 2. CUDA driver allocated memory was 2300575744 and is now 3076521984.
2025-12-04T14:26:42.1624458Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1624806Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1625420Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1625948Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1626311Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1626727Z [rank2]:E1204 14:04:37.518000 374345 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1627009Z dist init r=2, world=4
2025-12-04T14:26:42.1627212Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1627553Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1628040Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1628524Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1629006Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1629457Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1629894Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1630364Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1630828Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1631321Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1631786Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1632257Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1632711Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1633194Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1633881Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3026190336.
2025-12-04T14:26:42.1634524Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1634874Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1635490Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1636018Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1636384Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1636839Z [rank3]:E1204 14:04:37.566000 374346 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1637082Z dist init r=3, world=4
2025-12-04T14:26:42.1637487Z [rank0]:[W1204 14:04:37.138986692 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1637897Z FAILED [7.6136s] [100%]
2025-12-04T14:26:42.1637961Z 
2025-12-04T14:26:42.1638020Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1638234Z _ TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda _
2025-12-04T14:26:42.1638434Z Traceback (most recent call last):
2025-12-04T14:26:42.1638679Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1638922Z     self._join_processes(fn)
2025-12-04T14:26:42.1639165Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1639429Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1639693Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1639950Z     raise RuntimeError(error)
2025-12-04T14:26:42.1640099Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1640258Z Traceback (most recent call last):
2025-12-04T14:26:42.1640529Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1640791Z     getattr(self, test_name)()
2025-12-04T14:26:42.1641022Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1641251Z     fn()
2025-12-04T14:26:42.1641451Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1641697Z     method(*args, **kwargs)
2025-12-04T14:26:42.1641916Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1642141Z     method(*args, **kwargs)
2025-12-04T14:26:42.1642360Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1642607Z     with policy():
2025-12-04T14:26:42.1642921Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1643184Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1643652Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3229614080.
2025-12-04T14:26:42.1644082Z 
2025-12-04T14:26:42.1644187Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1644577Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1644889Z 
2025-12-04T14:26:42.1645003Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1645164Z 
2025-12-04T14:26:42.1645165Z 
2025-12-04T14:26:42.1645252Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1645491Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1645899Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-29ae8f388b4c215a.xml -
2025-12-04T14:26:42.1646263Z =========================== short test summary info ============================
2025-12-04T14:26:42.1646660Z FAILED [7.6136s] distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1646801Z Traceback (most recent call last):
2025-12-04T14:26:42.1646980Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1647047Z     getattr(self, test_name)()
2025-12-04T14:26:42.1647226Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1647281Z     fn()
2025-12-04T14:26:42.1647461Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1647529Z     method(*args, **kwargs)
2025-12-04T14:26:42.1647709Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1647762Z     method(*args, **kwargs)
2025-12-04T14:26:42.1647932Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1647993Z     with policy():
2025-12-04T14:26:42.1648213Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1648272Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1648668Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3229614080.
2025-12-04T14:26:42.1648687Z 
2025-12-04T14:26:42.1648775Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1649072Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestHooksCUDA.test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1649075Z 
2025-12-04T14:26:42.1649189Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1649278Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1649367Z ======================= 1 failed, 19 deselected in 7.77s =======================
2025-12-04T14:26:42.1649416Z Got exit code 1
2025-12-04T14:26:42.1649651Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda
2025-12-04T14:26:42.1649801Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.1650014Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4e6491e744eff92d.xml
2025-12-04T14:26:42.1650084Z ============================= test session starts ==============================
2025-12-04T14:26:42.1650220Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1650269Z cachedir: .pytest_cache
2025-12-04T14:26:42.1650478Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1650537Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1650605Z configfile: pytest.ini
2025-12-04T14:26:42.1650780Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1650876Z collecting ... collected 60 items / 2 deselected / 58 selected
2025-12-04T14:26:42.1650960Z stepcurrent: skipping 2 already run items.
2025-12-04T14:26:42.1651035Z Running 18 items in this shard
2025-12-04T14:26:42.1651038Z 
2025-12-04T14:26:42.1651362Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda I1204 14:04:41.535000 374660 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 374729
2025-12-04T14:26:42.1651541Z I1204 14:04:41.535000 374660 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 374730
2025-12-04T14:26:42.1651714Z I1204 14:04:41.536000 374660 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 374731
2025-12-04T14:26:42.1651895Z I1204 14:04:41.537000 374660 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 374732
2025-12-04T14:26:42.1652219Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1652282Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1652922Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1652983Z   _warn_cpu_init()
2025-12-04T14:26:42.1653304Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1653386Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1653697Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1653774Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1654358Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1654427Z   _warn_cpu_init()
2025-12-04T14:26:42.1655021Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1655071Z   _warn_cpu_init()
2025-12-04T14:26:42.1655401Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1655461Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1656061Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1656132Z   _warn_cpu_init()
2025-12-04T14:26:42.1656439Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1656544Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1656876Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1656980Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1657280Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1657385Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1657711Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1657809Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1659122Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1659297Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1660594Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1660744Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1662028Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1662198Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1662444Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1662512Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1662748Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1662831Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1663078Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1663158Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1664447Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1664613Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1664857Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1664925Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1665177Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1665231Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1665485Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1665548Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1665793Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1665861Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1666093Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1666163Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1666465Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1666536Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1666700Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1666939Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1667244Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1667424Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1667773Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1667917Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1668238Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1668398Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1668718Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1668873Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1669196Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1669361Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1669651Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1669828Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1670331Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 1. CUDA driver allocated memory was 2317352960 and is now 3877634048.
2025-12-04T14:26:42.1670498Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1670720Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1671101Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1671242Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1671462Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1671676Z [rank1]:E1204 14:05:12.512000 374730 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1671728Z dist init r=1, world=4
2025-12-04T14:26:42.1671893Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1672066Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1672378Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1672596Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1672895Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1673057Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1673365Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1673533Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1673837Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1674019Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1674326Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1674476Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1674776Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1674952Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1675467Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 2. CUDA driver allocated memory was 2300575744 and is now 3860856832.
2025-12-04T14:26:42.1675610Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1675817Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1676202Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1676335Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1676579Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1676876Z [rank2]:E1204 14:05:12.519000 374731 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1676945Z dist init r=2, world=4
2025-12-04T14:26:42.1677098Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1677286Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1677634Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1677814Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1678125Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1678274Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1678590Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1678753Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1679055Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1679224Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1679511Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1679682Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1679975Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1680148Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1680651Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 3. CUDA driver allocated memory was 2250244096 and is now 3810525184.
2025-12-04T14:26:42.1680776Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1681004Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1681384Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1681521Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1681744Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1681937Z [rank3]:E1204 14:05:12.553000 374732 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1682004Z dist init r=3, world=4
2025-12-04T14:26:42.1682181Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1682373Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1682676Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1682864Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1683155Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1683324Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1683612Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1683791Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1684094Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1684246Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1684561Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1684709Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1685017Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1685176Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1685675Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 0. CUDA driver allocated memory was 2453667840 and is now 4013948928.
2025-12-04T14:26:42.1685828Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1686043Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1686433Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1686557Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1686854Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1687059Z [rank0]:E1204 14:05:12.580000 374729 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1687121Z dist init r=0, world=4
2025-12-04T14:26:42.1687478Z [rank0]:[W1204 14:05:12.397528187 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1687546Z FAILED [33.0349s] [  5%]
2025-12-04T14:26:42.1687548Z 
2025-12-04T14:26:42.1687625Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1687749Z __ TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda ___
2025-12-04T14:26:42.1687829Z Traceback (most recent call last):
2025-12-04T14:26:42.1688006Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1688073Z     self._join_processes(fn)
2025-12-04T14:26:42.1688257Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1688346Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1688541Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1688612Z     raise RuntimeError(error)
2025-12-04T14:26:42.1688718Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.1688780Z Traceback (most recent call last):
2025-12-04T14:26:42.1688972Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1689035Z     getattr(self, test_name)()
2025-12-04T14:26:42.1689220Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1689270Z     fn()
2025-12-04T14:26:42.1689453Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1689502Z     method(*args, **kwargs)
2025-12-04T14:26:42.1689691Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1689744Z     method(*args, **kwargs)
2025-12-04T14:26:42.1689921Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1689975Z     with policy():
2025-12-04T14:26:42.1690149Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1690213Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1690606Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 1. CUDA driver allocated memory was 2317352960 and is now 3877634048.
2025-12-04T14:26:42.1690609Z 
2025-12-04T14:26:42.1690717Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1690970Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1690973Z 
2025-12-04T14:26:42.1691083Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1691085Z 
2025-12-04T14:26:42.1691169Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1691249Z Traceback (most recent call last):
2025-12-04T14:26:42.1691456Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1691526Z     getattr(self, test_name)()
2025-12-04T14:26:42.1691713Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1691775Z     fn()
2025-12-04T14:26:42.1691941Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1692028Z     method(*args, **kwargs)
2025-12-04T14:26:42.1692188Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1692254Z     method(*args, **kwargs)
2025-12-04T14:26:42.1692410Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1692494Z     with policy():
2025-12-04T14:26:42.1692672Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1692728Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1693109Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 2. CUDA driver allocated memory was 2300575744 and is now 3860856832.
2025-12-04T14:26:42.1693113Z 
2025-12-04T14:26:42.1693192Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1693468Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1693470Z 
2025-12-04T14:26:42.1693569Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1693572Z 
2025-12-04T14:26:42.1693655Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1693712Z Traceback (most recent call last):
2025-12-04T14:26:42.1693895Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1693982Z     getattr(self, test_name)()
2025-12-04T14:26:42.1694152Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1694215Z     fn()
2025-12-04T14:26:42.1694380Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1694440Z     method(*args, **kwargs)
2025-12-04T14:26:42.1694615Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1694684Z     method(*args, **kwargs)
2025-12-04T14:26:42.1694847Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1694909Z     with policy():
2025-12-04T14:26:42.1695071Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1695145Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1695514Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 3. CUDA driver allocated memory was 2250244096 and is now 3810525184.
2025-12-04T14:26:42.1695518Z 
2025-12-04T14:26:42.1695613Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1695856Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1695893Z 
2025-12-04T14:26:42.1695995Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1695997Z 
2025-12-04T14:26:42.1696010Z 
2025-12-04T14:26:42.1696117Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1696221Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1696480Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4e6491e744eff92d.xml -
2025-12-04T14:26:42.1696570Z =========================== short test summary info ============================
2025-12-04T14:26:42.1696888Z FAILED [33.0349s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.1696940Z Traceback (most recent call last):
2025-12-04T14:26:42.1697150Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1697203Z     getattr(self, test_name)()
2025-12-04T14:26:42.1697396Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1697442Z     fn()
2025-12-04T14:26:42.1697613Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1697698Z     method(*args, **kwargs)
2025-12-04T14:26:42.1697861Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1697931Z     method(*args, **kwargs)
2025-12-04T14:26:42.1698092Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1698146Z     with policy():
2025-12-04T14:26:42.1698318Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1698387Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1698755Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 1. CUDA driver allocated memory was 2317352960 and is now 3877634048.
2025-12-04T14:26:42.1698758Z 
2025-12-04T14:26:42.1698854Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1699096Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1699098Z 
2025-12-04T14:26:42.1699213Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1699215Z 
2025-12-04T14:26:42.1699311Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1699367Z Traceback (most recent call last):
2025-12-04T14:26:42.1699555Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1699608Z     getattr(self, test_name)()
2025-12-04T14:26:42.1717071Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1717123Z     fn()
2025-12-04T14:26:42.1717301Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1717344Z     method(*args, **kwargs)
2025-12-04T14:26:42.1717501Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1717541Z     method(*args, **kwargs)
2025-12-04T14:26:42.1717755Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1717795Z     with policy():
2025-12-04T14:26:42.1717965Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1718007Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1718372Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 2. CUDA driver allocated memory was 2300575744 and is now 3860856832.
2025-12-04T14:26:42.1718391Z 
2025-12-04T14:26:42.1718470Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1718712Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1718717Z 
2025-12-04T14:26:42.1718809Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1718812Z 
2025-12-04T14:26:42.1718873Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1718923Z Traceback (most recent call last):
2025-12-04T14:26:42.1719091Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1719135Z     getattr(self, test_name)()
2025-12-04T14:26:42.1719298Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1719333Z     fn()
2025-12-04T14:26:42.1719482Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1719522Z     method(*args, **kwargs)
2025-12-04T14:26:42.1719673Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1719713Z     method(*args, **kwargs)
2025-12-04T14:26:42.1719861Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1719901Z     with policy():
2025-12-04T14:26:42.1720052Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1720093Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1720449Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 3. CUDA driver allocated memory was 2250244096 and is now 3810525184.
2025-12-04T14:26:42.1720453Z 
2025-12-04T14:26:42.1720528Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1720767Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1720771Z 
2025-12-04T14:26:42.1720859Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1720926Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1720989Z ======================= 1 failed, 2 deselected in 33.20s =======================
2025-12-04T14:26:42.1721031Z Got exit code 1
2025-12-04T14:26:42.1721071Z Retrying single test...
2025-12-04T14:26:42.1721266Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-ba1a0061c9045eb7.xml
2025-12-04T14:26:42.1721324Z ============================= test session starts ==============================
2025-12-04T14:26:42.1721442Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1724585Z cachedir: .pytest_cache
2025-12-04T14:26:42.1724750Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1724809Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1724851Z configfile: pytest.ini
2025-12-04T14:26:42.1725015Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1725108Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1725337Z stepcurrent: skipping 2 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1725383Z Running 1 items in this shard
2025-12-04T14:26:42.1725386Z 
2025-12-04T14:26:42.1725701Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda I1204 14:05:16.804000 375062 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 375131
2025-12-04T14:26:42.1725859Z I1204 14:05:16.805000 375062 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 375132
2025-12-04T14:26:42.1726015Z I1204 14:05:16.806000 375062 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 375133
2025-12-04T14:26:42.1726165Z I1204 14:05:16.807000 375062 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 375134
2025-12-04T14:26:42.1726464Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1726517Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1726861Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1726912Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1727494Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1727536Z   _warn_cpu_init()
2025-12-04T14:26:42.1728105Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1728146Z   _warn_cpu_init()
2025-12-04T14:26:42.1728436Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1728487Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1728773Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1728819Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1729416Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1729466Z   _warn_cpu_init()
2025-12-04T14:26:42.1730031Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1730086Z   _warn_cpu_init()
2025-12-04T14:26:42.1730375Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1730455Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1730743Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1730824Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1731108Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1731183Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1731466Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1731541Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1732824Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1732951Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1733187Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1733231Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1734514Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1734668Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1735928Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1736051Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1736279Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1736326Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1736549Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1736592Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1737891Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1738014Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1738242Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1738283Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1738533Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1738593Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1738813Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1738853Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1739087Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1739126Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1739347Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1739388Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1739683Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1739725Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1739873Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1740039Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1740329Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1740488Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1740774Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1740903Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1741181Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1741336Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1741615Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1741764Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1742043Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1742180Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1742459Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1742625Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1743116Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 0. CUDA driver allocated memory was 2453667840 and is now 4013948928.
2025-12-04T14:26:42.1743256Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1743452Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1743821Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1743936Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1744152Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1744317Z [rank0]:E1204 14:05:47.752000 375131 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1744358Z dist init r=0, world=4
2025-12-04T14:26:42.1744498Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1744658Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1744949Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1745104Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1745393Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1745518Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1745798Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1745945Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1746223Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1746371Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1746647Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1746826Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1747130Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1747293Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1747778Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 3. CUDA driver allocated memory was 2250244096 and is now 3810525184.
2025-12-04T14:26:42.1747910Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1748108Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1748468Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1748583Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1748794Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1748960Z [rank3]:E1204 14:05:47.762000 375134 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1749099Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1749260Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1749551Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1749705Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1749991Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1750115Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1750395Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1750542Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1750821Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1750971Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1751273Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1751410Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1751700Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1751862Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1752341Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 2. CUDA driver allocated memory was 2300575744 and is now 3860856832.
2025-12-04T14:26:42.1752456Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1752654Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1753011Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1753128Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1753341Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1753507Z [rank2]:E1204 14:05:47.762000 375133 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1753547Z dist init r=3, world=4
2025-12-04T14:26:42.1753588Z dist init r=2, world=4
2025-12-04T14:26:42.1753725Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1753885Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1754174Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1754326Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1754612Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1754735Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1755013Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1755161Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1755460Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1755609Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1755893Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1756031Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1756317Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1756466Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1756974Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 1. CUDA driver allocated memory was 2317352960 and is now 3877634048.
2025-12-04T14:26:42.1757092Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1757290Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1757653Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1757768Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1757980Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1758147Z [rank1]:E1204 14:05:47.770000 375132 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1758187Z dist init r=1, world=4
2025-12-04T14:26:42.1758529Z [rank0]:[W1204 14:05:47.412968571 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1758570Z FAILED [32.7355s] [100%]
2025-12-04T14:26:42.1758575Z 
2025-12-04T14:26:42.1758637Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1758741Z __ TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda ___
2025-12-04T14:26:42.1758789Z Traceback (most recent call last):
2025-12-04T14:26:42.1758952Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1758995Z     self._join_processes(fn)
2025-12-04T14:26:42.1759168Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1759223Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1759402Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1759447Z     raise RuntimeError(error)
2025-12-04T14:26:42.1759557Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1759602Z Traceback (most recent call last):
2025-12-04T14:26:42.1759766Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1759821Z     getattr(self, test_name)()
2025-12-04T14:26:42.1759980Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1760014Z     fn()
2025-12-04T14:26:42.1760180Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1760220Z     method(*args, **kwargs)
2025-12-04T14:26:42.1760372Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1760411Z     method(*args, **kwargs)
2025-12-04T14:26:42.1760563Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1760599Z     with policy():
2025-12-04T14:26:42.1760752Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1760792Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1761150Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 0. CUDA driver allocated memory was 2453667840 and is now 4013948928.
2025-12-04T14:26:42.1761153Z 
2025-12-04T14:26:42.1761231Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1761464Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1761466Z 
2025-12-04T14:26:42.1761555Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1761557Z 
2025-12-04T14:26:42.1761559Z 
2025-12-04T14:26:42.1761636Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1761725Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1761960Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-ba1a0061c9045eb7.xml -
2025-12-04T14:26:42.1762021Z =========================== short test summary info ============================
2025-12-04T14:26:42.1762274Z FAILED [32.7355s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1762321Z Traceback (most recent call last):
2025-12-04T14:26:42.1762486Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1762527Z     getattr(self, test_name)()
2025-12-04T14:26:42.1762688Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1762723Z     fn()
2025-12-04T14:26:42.1762875Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1762914Z     method(*args, **kwargs)
2025-12-04T14:26:42.1763070Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1763109Z     method(*args, **kwargs)
2025-12-04T14:26:42.1763260Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1763297Z     with policy():
2025-12-04T14:26:42.1763471Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1763512Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1763876Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 0. CUDA driver allocated memory was 2453667840 and is now 4013948928.
2025-12-04T14:26:42.1763879Z 
2025-12-04T14:26:42.1763964Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1764200Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1764202Z 
2025-12-04T14:26:42.1764291Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1764355Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1764420Z ====================== 1 failed, 19 deselected in 32.89s =======================
2025-12-04T14:26:42.1764458Z Got exit code 1
2025-12-04T14:26:42.1764500Z Retrying single test...
2025-12-04T14:26:42.1764687Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-c7a814f399e4285f.xml
2025-12-04T14:26:42.1764745Z ============================= test session starts ==============================
2025-12-04T14:26:42.1764860Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1764902Z cachedir: .pytest_cache
2025-12-04T14:26:42.1765059Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1765107Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1765146Z configfile: pytest.ini
2025-12-04T14:26:42.1765312Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1765387Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1765616Z stepcurrent: skipping 2 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1765660Z Running 1 items in this shard
2025-12-04T14:26:42.1765662Z 
2025-12-04T14:26:42.1765972Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda I1204 14:05:51.997000 375464 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 375533
2025-12-04T14:26:42.1766127Z I1204 14:05:51.998000 375464 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 375534
2025-12-04T14:26:42.1766280Z I1204 14:05:51.999000 375464 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 375535
2025-12-04T14:26:42.1766432Z I1204 14:05:51.999000 375464 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 375536
2025-12-04T14:26:42.1766724Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1766814Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1767425Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1767467Z   _warn_cpu_init()
2025-12-04T14:26:42.1767754Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1767820Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1768106Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1768168Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1768451Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1768499Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1769071Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1769109Z   _warn_cpu_init()
2025-12-04T14:26:42.1769676Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1769713Z   _warn_cpu_init()
2025-12-04T14:26:42.1770282Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1770321Z   _warn_cpu_init()
2025-12-04T14:26:42.1770607Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1770687Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1770971Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1771048Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1771332Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1771406Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1771689Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1771760Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1773060Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1773209Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1773437Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1773484Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1774746Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1774870Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1776129Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1776252Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1776479Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1776525Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1776821Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1776879Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1778140Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.1778276Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.1778503Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1778549Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1778773Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1778815Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1779037Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1779082Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1779301Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1779343Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1779562Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1779605Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1779897Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1779939Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1780085Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1780250Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1780542Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1780702Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1780986Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1781136Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1781415Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1781575Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1781869Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1782017Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1782297Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1782434Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1782712Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1782861Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1783345Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 0. CUDA driver allocated memory was 2453667840 and is now 4013948928.
2025-12-04T14:26:42.1783463Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1783660Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1784023Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1784139Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1784356Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1784522Z [rank0]:E1204 14:06:22.926000 375533 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1784662Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1784821Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1785109Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1785263Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1785564Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1785700Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1785977Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1786138Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1786414Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1786565Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1786886Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1787023Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1787302Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1787450Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1787931Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 3. CUDA driver allocated memory was 2250244096 and is now 3810525184.
2025-12-04T14:26:42.1788046Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1788243Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1788608Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1788721Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1788937Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1789101Z [rank3]:E1204 14:06:22.926000 375536 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1789142Z dist init r=0, world=4
2025-12-04T14:26:42.1789180Z dist init r=3, world=4
2025-12-04T14:26:42.1789319Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1789477Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1789791Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1789957Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1790241Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1790382Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1790659Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1790808Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1791084Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1791236Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1791515Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1791649Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1791929Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1792078Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1792558Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 1. CUDA driver allocated memory was 2317352960 and is now 3877634048.
2025-12-04T14:26:42.1792672Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1792869Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1793229Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1793342Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1793559Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1793722Z [rank1]:E1204 14:06:22.962000 375534 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1793762Z dist init r=1, world=4
2025-12-04T14:26:42.1793919Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1794080Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1794376Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1794529Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1794824Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1794951Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1795228Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1795375Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1795653Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1795801Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1796078Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1796215Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1796492Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1796641Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1797158Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 2. CUDA driver allocated memory was 2300575744 and is now 3860856832.
2025-12-04T14:26:42.1797273Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1797468Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1797828Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1797942Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1798153Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1798345Z [rank2]:E1204 14:06:23.028000 375535 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1798410Z dist init r=2, world=4
2025-12-04T14:26:42.1798747Z [rank0]:[W1204 14:06:23.588743491 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1798800Z FAILED [32.7371s] [100%]
2025-12-04T14:26:42.1798802Z 
2025-12-04T14:26:42.1798860Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1798961Z __ TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda ___
2025-12-04T14:26:42.1799009Z Traceback (most recent call last):
2025-12-04T14:26:42.1799172Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1799217Z     self._join_processes(fn)
2025-12-04T14:26:42.1799389Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1799447Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1799625Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1799669Z     raise RuntimeError(error)
2025-12-04T14:26:42.1799750Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1799794Z Traceback (most recent call last):
2025-12-04T14:26:42.1799956Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1799998Z     getattr(self, test_name)()
2025-12-04T14:26:42.1800158Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1800193Z     fn()
2025-12-04T14:26:42.1800344Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1800386Z     method(*args, **kwargs)
2025-12-04T14:26:42.1800541Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1800581Z     method(*args, **kwargs)
2025-12-04T14:26:42.1800735Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1800772Z     with policy():
2025-12-04T14:26:42.1800924Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1800964Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1801320Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 0. CUDA driver allocated memory was 2453667840 and is now 4013948928.
2025-12-04T14:26:42.1801323Z 
2025-12-04T14:26:42.1801398Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1801633Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1801636Z 
2025-12-04T14:26:42.1801725Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1801727Z 
2025-12-04T14:26:42.1801729Z 
2025-12-04T14:26:42.1801803Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1801892Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1802148Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-c7a814f399e4285f.xml -
2025-12-04T14:26:42.1802219Z =========================== short test summary info ============================
2025-12-04T14:26:42.1802469Z FAILED [32.7371s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1802517Z Traceback (most recent call last):
2025-12-04T14:26:42.1802691Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1802734Z     getattr(self, test_name)()
2025-12-04T14:26:42.1802893Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1802929Z     fn()
2025-12-04T14:26:42.1803080Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1803121Z     method(*args, **kwargs)
2025-12-04T14:26:42.1803273Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1803312Z     method(*args, **kwargs)
2025-12-04T14:26:42.1803465Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1803502Z     with policy():
2025-12-04T14:26:42.1803654Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1803694Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1804054Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 147968 on device 0. CUDA driver allocated memory was 2453667840 and is now 4013948928.
2025-12-04T14:26:42.1804056Z 
2025-12-04T14:26:42.1804130Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1804367Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1804369Z 
2025-12-04T14:26:42.1804455Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1804519Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1804582Z ====================== 1 failed, 19 deselected in 32.90s =======================
2025-12-04T14:26:42.1804620Z Got exit code 1
2025-12-04T14:26:42.1804800Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda
2025-12-04T14:26:42.1804936Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.1805128Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-557a24ab18b8ed8e.xml
2025-12-04T14:26:42.1805187Z ============================= test session starts ==============================
2025-12-04T14:26:42.1805301Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1805343Z cachedir: .pytest_cache
2025-12-04T14:26:42.1805502Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1805549Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1805590Z configfile: pytest.ini
2025-12-04T14:26:42.1805750Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1805850Z collecting ... collected 60 items / 3 deselected / 57 selected
2025-12-04T14:26:42.1805903Z stepcurrent: skipping 3 already run items.
2025-12-04T14:26:42.1805948Z Running 17 items in this shard
2025-12-04T14:26:42.1805963Z 
2025-12-04T14:26:42.1806272Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda I1204 14:06:27.332000 375866 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 375935
2025-12-04T14:26:42.1806430Z I1204 14:06:27.333000 375866 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 375936
2025-12-04T14:26:42.1806594Z I1204 14:06:27.333000 375866 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 375937
2025-12-04T14:26:42.1806778Z I1204 14:06:27.334000 375866 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 375938
2025-12-04T14:26:42.1807073Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1807123Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1807413Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1807461Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1807746Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1807793Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1808077Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1808125Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1808704Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1808744Z   _warn_cpu_init()
2025-12-04T14:26:42.1809312Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1809351Z   _warn_cpu_init()
2025-12-04T14:26:42.1809920Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1809960Z   _warn_cpu_init()
2025-12-04T14:26:42.1810553Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1810605Z   _warn_cpu_init()
2025-12-04T14:26:42.1810893Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1810989Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1811275Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1811352Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1811637Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1811712Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1811995Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1812069Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1812362Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1812406Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1812635Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1812678Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1812901Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1812944Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1813165Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1813207Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1813428Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1813470Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1813690Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1813730Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1813950Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1813990Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1814211Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1814252Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1814499Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1814539Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1814696Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1814858Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1815151Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1815319Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1815607Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1815733Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1816010Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1816160Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1816438Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1816590Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1816910Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1817050Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1817328Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1817476Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1817962Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 1. CUDA driver allocated memory was 2317352960 and is now 3854565376.
2025-12-04T14:26:42.1818078Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1818275Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1818636Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1818778Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1818993Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1819171Z [rank1]:E1204 14:07:04.448000 375936 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1819212Z dist init r=1, world=4
2025-12-04T14:26:42.1819350Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1819526Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1819814Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1819969Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1820255Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1820379Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1820658Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1820805Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1821083Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1821231Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1821508Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1821644Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1821924Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1822073Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1822551Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 2. CUDA driver allocated memory was 2300575744 and is now 3837788160.
2025-12-04T14:26:42.1822667Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1822861Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1823240Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1823367Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1823581Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1823761Z [rank2]:E1204 14:07:04.457000 375937 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1823800Z dist init r=2, world=4
2025-12-04T14:26:42.1823939Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1824099Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1824388Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1824542Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1824829Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1824955Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1825231Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1825380Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1825654Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1825804Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1826079Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1826217Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1826494Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1826642Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1827160Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 3. CUDA driver allocated memory was 2250244096 and is now 3787456512.
2025-12-04T14:26:42.1827300Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1827498Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1827868Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1827996Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1828210Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1828375Z [rank3]:E1204 14:07:04.511000 375938 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1828414Z dist init r=3, world=4
2025-12-04T14:26:42.1828552Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1828713Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1828998Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1829153Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1829438Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1829563Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1829841Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1829987Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1830264Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1830411Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1830689Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1830825Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1831102Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1831252Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1831750Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 0. CUDA driver allocated memory was 2453667840 and is now 3990880256.
2025-12-04T14:26:42.1831877Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1832071Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1832442Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1832553Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1832768Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1832934Z [rank0]:E1204 14:07:04.518000 375935 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1832972Z dist init r=0, world=4
2025-12-04T14:26:42.1833310Z [rank0]:[W1204 14:07:04.277181603 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1833351Z FAILED [39.0400s] [  5%]
2025-12-04T14:26:42.1833353Z 
2025-12-04T14:26:42.1833409Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1833510Z ___ TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda ___
2025-12-04T14:26:42.1833561Z Traceback (most recent call last):
2025-12-04T14:26:42.1833723Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1833769Z     self._join_processes(fn)
2025-12-04T14:26:42.1833941Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1833996Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1834173Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1834218Z     raise RuntimeError(error)
2025-12-04T14:26:42.1834297Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.1834343Z Traceback (most recent call last):
2025-12-04T14:26:42.1834506Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1834549Z     getattr(self, test_name)()
2025-12-04T14:26:42.1834708Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1834744Z     fn()
2025-12-04T14:26:42.1834895Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1834935Z     method(*args, **kwargs)
2025-12-04T14:26:42.1835087Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1835127Z     method(*args, **kwargs)
2025-12-04T14:26:42.1835279Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1835315Z     with policy():
2025-12-04T14:26:42.1835498Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1835540Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1835891Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 1. CUDA driver allocated memory was 2317352960 and is now 3854565376.
2025-12-04T14:26:42.1835906Z 
2025-12-04T14:26:42.1835981Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1836228Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1836230Z 
2025-12-04T14:26:42.1836317Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1836321Z 
2025-12-04T14:26:42.1836380Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1836427Z Traceback (most recent call last):
2025-12-04T14:26:42.1836589Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1836634Z     getattr(self, test_name)()
2025-12-04T14:26:42.1836828Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1836864Z     fn()
2025-12-04T14:26:42.1837014Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1837056Z     method(*args, **kwargs)
2025-12-04T14:26:42.1837205Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1837247Z     method(*args, **kwargs)
2025-12-04T14:26:42.1837397Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1837438Z     with policy():
2025-12-04T14:26:42.1837588Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1837631Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1837981Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 2. CUDA driver allocated memory was 2300575744 and is now 3837788160.
2025-12-04T14:26:42.1837984Z 
2025-12-04T14:26:42.1838058Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1838291Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1838295Z 
2025-12-04T14:26:42.1838383Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1838385Z 
2025-12-04T14:26:42.1838387Z 
2025-12-04T14:26:42.1838463Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1838551Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1838786Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-557a24ab18b8ed8e.xml -
2025-12-04T14:26:42.1838847Z =========================== short test summary info ============================
2025-12-04T14:26:42.1839098Z FAILED [39.0400s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.1839143Z Traceback (most recent call last):
2025-12-04T14:26:42.1839335Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1839377Z     getattr(self, test_name)()
2025-12-04T14:26:42.1839537Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1839585Z     fn()
2025-12-04T14:26:42.1839737Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1839780Z     method(*args, **kwargs)
2025-12-04T14:26:42.1839948Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1839992Z     method(*args, **kwargs)
2025-12-04T14:26:42.1840144Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1840184Z     with policy():
2025-12-04T14:26:42.1840337Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1840381Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1840734Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 1. CUDA driver allocated memory was 2317352960 and is now 3854565376.
2025-12-04T14:26:42.1840738Z 
2025-12-04T14:26:42.1840815Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1841046Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1841048Z 
2025-12-04T14:26:42.1841138Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1841140Z 
2025-12-04T14:26:42.1841201Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1841250Z Traceback (most recent call last):
2025-12-04T14:26:42.1841413Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1841460Z     getattr(self, test_name)()
2025-12-04T14:26:42.1841619Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1841656Z     fn()
2025-12-04T14:26:42.1841810Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1841851Z     method(*args, **kwargs)
2025-12-04T14:26:42.1842007Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1842047Z     method(*args, **kwargs)
2025-12-04T14:26:42.1842199Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1842237Z     with policy():
2025-12-04T14:26:42.1842392Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1842434Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1842787Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 2. CUDA driver allocated memory was 2300575744 and is now 3837788160.
2025-12-04T14:26:42.1842789Z 
2025-12-04T14:26:42.1842862Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1843098Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1843100Z 
2025-12-04T14:26:42.1843207Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1843274Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1843346Z ======================= 1 failed, 3 deselected in 39.20s =======================
2025-12-04T14:26:42.1843387Z Got exit code 1
2025-12-04T14:26:42.1843428Z Retrying single test...
2025-12-04T14:26:42.1843620Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d06c0728eae6a551.xml
2025-12-04T14:26:42.1843698Z ============================= test session starts ==============================
2025-12-04T14:26:42.1843812Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1843857Z cachedir: .pytest_cache
2025-12-04T14:26:42.1844016Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1844065Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1844108Z configfile: pytest.ini
2025-12-04T14:26:42.1844273Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1844347Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1844574Z stepcurrent: skipping 3 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1844620Z Running 1 items in this shard
2025-12-04T14:26:42.1844622Z 
2025-12-04T14:26:42.1844930Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda I1204 14:07:08.795000 376268 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 376337
2025-12-04T14:26:42.1845085Z I1204 14:07:08.796000 376268 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 376338
2025-12-04T14:26:42.1845240Z I1204 14:07:08.796000 376268 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 376339
2025-12-04T14:26:42.1845393Z I1204 14:07:08.797000 376268 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 376340
2025-12-04T14:26:42.1845684Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1845740Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1846026Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1846077Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1846655Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1846697Z   _warn_cpu_init()
2025-12-04T14:26:42.1847303Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1847374Z   _warn_cpu_init()
2025-12-04T14:26:42.1847664Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1847725Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1848011Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1848072Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1848648Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1848689Z   _warn_cpu_init()
2025-12-04T14:26:42.1849252Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1849293Z   _warn_cpu_init()
2025-12-04T14:26:42.1849581Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1849664Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1849950Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1850031Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1850315Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1850393Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1850681Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1850754Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1851050Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1851094Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1851327Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1851370Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1851597Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1851638Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1851883Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1851936Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1852158Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1852202Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1852434Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1852478Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1852697Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1852743Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1852964Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1853009Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1853228Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1853272Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1853420Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1853587Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1853879Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1854037Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1854326Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1854453Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1854733Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1854883Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1855163Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1855312Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1855591Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1855733Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1856034Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1856185Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1856675Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 2. CUDA driver allocated memory was 2300575744 and is now 3837788160.
2025-12-04T14:26:42.1857003Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1857200Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1857568Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1857686Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1857898Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1858067Z [rank2]:E1204 14:07:45.973000 376339 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1858107Z dist init r=2, world=4
2025-12-04T14:26:42.1858249Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1858412Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1858702Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1858856Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1859146Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1859273Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1859550Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1859701Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1859977Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1860128Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1860434Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1860575Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1860876Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1861025Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1861522Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 1. CUDA driver allocated memory was 2317352960 and is now 3854565376.
2025-12-04T14:26:42.1861638Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1861837Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1862197Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1862314Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1862529Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1862695Z [rank1]:E1204 14:07:45.980000 376338 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1862740Z dist init r=1, world=4
2025-12-04T14:26:42.1862880Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1863045Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1863332Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1863490Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1863776Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1863903Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1864184Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1864333Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1864612Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1864779Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1865057Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1865206Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1865488Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1865651Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1866126Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 3. CUDA driver allocated memory was 2250244096 and is now 3787456512.
2025-12-04T14:26:42.1866245Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1866440Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1866845Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1866962Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1867173Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1867340Z [rank3]:E1204 14:07:45.985000 376340 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1867379Z dist init r=3, world=4
2025-12-04T14:26:42.1867521Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1867683Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1867974Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1868127Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1868414Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1868537Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1868818Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1868967Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1869269Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1869432Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1869708Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1869864Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1870148Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1870303Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1870780Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 0. CUDA driver allocated memory was 2453667840 and is now 3990880256.
2025-12-04T14:26:42.1870894Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1871091Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1871453Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1871571Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1871785Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1871949Z [rank0]:E1204 14:07:46.029000 376337 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1871991Z dist init r=0, world=4
2025-12-04T14:26:42.1872329Z [rank0]:[W1204 14:07:46.792309127 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1872373Z FAILED [39.0421s] [100%]
2025-12-04T14:26:42.1872375Z 
2025-12-04T14:26:42.1872433Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1872538Z ___ TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda ___
2025-12-04T14:26:42.1872585Z Traceback (most recent call last):
2025-12-04T14:26:42.1872752Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1872798Z     self._join_processes(fn)
2025-12-04T14:26:42.1872974Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1873029Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1873231Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1873276Z     raise RuntimeError(error)
2025-12-04T14:26:42.1873359Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1873423Z Traceback (most recent call last):
2025-12-04T14:26:42.1873587Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1873630Z     getattr(self, test_name)()
2025-12-04T14:26:42.1873790Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1873840Z     fn()
2025-12-04T14:26:42.1873991Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1874037Z     method(*args, **kwargs)
2025-12-04T14:26:42.1874191Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1874234Z     method(*args, **kwargs)
2025-12-04T14:26:42.1874384Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1874427Z     with policy():
2025-12-04T14:26:42.1874578Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1874623Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1874978Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 3. CUDA driver allocated memory was 2250244096 and is now 3787456512.
2025-12-04T14:26:42.1874981Z 
2025-12-04T14:26:42.1875059Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1875292Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1875295Z 
2025-12-04T14:26:42.1875385Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1875387Z 
2025-12-04T14:26:42.1875389Z 
2025-12-04T14:26:42.1875466Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1875555Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1875792Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d06c0728eae6a551.xml -
2025-12-04T14:26:42.1875852Z =========================== short test summary info ============================
2025-12-04T14:26:42.1876106Z FAILED [39.0421s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.1876152Z Traceback (most recent call last):
2025-12-04T14:26:42.1876318Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1876362Z     getattr(self, test_name)()
2025-12-04T14:26:42.1876523Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1876558Z     fn()
2025-12-04T14:26:42.1876711Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1876785Z     method(*args, **kwargs)
2025-12-04T14:26:42.1876939Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1876978Z     method(*args, **kwargs)
2025-12-04T14:26:42.1877164Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1877203Z     with policy():
2025-12-04T14:26:42.1877357Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1877413Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1877765Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 3. CUDA driver allocated memory was 2250244096 and is now 3787456512.
2025-12-04T14:26:42.1877780Z 
2025-12-04T14:26:42.1877857Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1878090Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1878091Z 
2025-12-04T14:26:42.1878182Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1878245Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1878312Z ====================== 1 failed, 19 deselected in 39.21s =======================
2025-12-04T14:26:42.1878349Z Got exit code 1
2025-12-04T14:26:42.1878392Z Retrying single test...
2025-12-04T14:26:42.1878582Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-bce3754b9a2ca69b.xml
2025-12-04T14:26:42.1878643Z ============================= test session starts ==============================
2025-12-04T14:26:42.1878755Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1878799Z cachedir: .pytest_cache
2025-12-04T14:26:42.1878956Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1879007Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1879047Z configfile: pytest.ini
2025-12-04T14:26:42.1879211Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1879290Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1879515Z stepcurrent: skipping 3 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1879563Z Running 1 items in this shard
2025-12-04T14:26:42.1879565Z 
2025-12-04T14:26:42.1879871Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda I1204 14:07:50.262000 376670 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 376739
2025-12-04T14:26:42.1880031Z I1204 14:07:50.263000 376670 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 376740
2025-12-04T14:26:42.1880183Z I1204 14:07:50.263000 376670 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 376741
2025-12-04T14:26:42.1880337Z I1204 14:07:50.264000 376670 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 376742
2025-12-04T14:26:42.1880627Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1880681Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1880969Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1881018Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1881324Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1881383Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1881673Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1881732Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.1882316Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1882355Z   _warn_cpu_init()
2025-12-04T14:26:42.1882926Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1882968Z   _warn_cpu_init()
2025-12-04T14:26:42.1883535Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1883577Z   _warn_cpu_init()
2025-12-04T14:26:42.1884142Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1884185Z   _warn_cpu_init()
2025-12-04T14:26:42.1884477Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1884555Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1884846Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1884921Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1885209Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1885281Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1885589Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.1885662Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.1885964Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1886011Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1886240Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1886297Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1886522Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1886565Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1886828Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1886872Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1887091Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.1887134Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1887352Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1887395Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1887614Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1887657Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1887878Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1887921Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1888144Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.1888185Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1888332Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1888495Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1888789Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1888947Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1889235Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1889361Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1889640Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1889823Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1890099Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1890265Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1890554Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1890694Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1890973Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1891126Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1891611Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 0. CUDA driver allocated memory was 2453667840 and is now 3990880256.
2025-12-04T14:26:42.1891727Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1891927Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1892286Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1892404Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1892619Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1892784Z [rank0]:E1204 14:08:27.454000 376739 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1892828Z dist init r=0, world=4
2025-12-04T14:26:42.1892967Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1893129Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1893419Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1893579Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1893866Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1894017Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1894294Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1894454Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1894736Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1894896Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1895174Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1895310Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1895590Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1895738Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1896222Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 3. CUDA driver allocated memory was 2250244096 and is now 3787456512.
2025-12-04T14:26:42.1896339Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1896535Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1896933Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1897046Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1897259Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1897425Z [rank3]:E1204 14:08:27.470000 376742 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1897465Z dist init r=3, world=4
2025-12-04T14:26:42.1897603Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1897763Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1898052Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1898205Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1898518Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1898655Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1898934Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1899098Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1899374Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1899522Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1899798Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1899935Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1900213Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1900363Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1900841Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 2. CUDA driver allocated memory was 2300575744 and is now 3837788160.
2025-12-04T14:26:42.1900956Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1901154Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1901511Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1901626Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1901837Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1902003Z [rank2]:E1204 14:08:27.484000 376741 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1902041Z dist init r=2, world=4
2025-12-04T14:26:42.1902181Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1902341Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1902649Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1902804Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1903099Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1903236Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1903511Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1903661Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1903938Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1904085Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1904361Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1904497Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1904777Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1904925Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1905402Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 1. CUDA driver allocated memory was 2317352960 and is now 3854565376.
2025-12-04T14:26:42.1905518Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1905713Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1906073Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1906185Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1906397Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1906561Z [rank1]:E1204 14:08:27.510000 376740 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1906601Z dist init r=1, world=4
2025-12-04T14:26:42.1907005Z [rank0]:[W1204 14:08:27.107219087 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1907046Z FAILED [38.9410s] [100%]
2025-12-04T14:26:42.1907061Z 
2025-12-04T14:26:42.1907119Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1907219Z ___ TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda ___
2025-12-04T14:26:42.1907267Z Traceback (most recent call last):
2025-12-04T14:26:42.1907455Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1907502Z     self._join_processes(fn)
2025-12-04T14:26:42.1907673Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1907727Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1907905Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1907949Z     raise RuntimeError(error)
2025-12-04T14:26:42.1908029Z RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1908075Z Traceback (most recent call last):
2025-12-04T14:26:42.1908235Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1908279Z     getattr(self, test_name)()
2025-12-04T14:26:42.1908437Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1908474Z     fn()
2025-12-04T14:26:42.1908624Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1908667Z     method(*args, **kwargs)
2025-12-04T14:26:42.1908819Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1908860Z     method(*args, **kwargs)
2025-12-04T14:26:42.1909010Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1909053Z     with policy():
2025-12-04T14:26:42.1909205Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1909246Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1909597Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 2. CUDA driver allocated memory was 2300575744 and is now 3837788160.
2025-12-04T14:26:42.1909599Z 
2025-12-04T14:26:42.1909674Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1909909Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1909912Z 
2025-12-04T14:26:42.1909999Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1910001Z 
2025-12-04T14:26:42.1910003Z 
2025-12-04T14:26:42.1910081Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1910169Z Process 2 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1910405Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-bce3754b9a2ca69b.xml -
2025-12-04T14:26:42.1910467Z =========================== short test summary info ============================
2025-12-04T14:26:42.1910741Z FAILED [38.9410s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda - RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1910790Z Traceback (most recent call last):
2025-12-04T14:26:42.1910965Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1911010Z     getattr(self, test_name)()
2025-12-04T14:26:42.1911169Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1911220Z     fn()
2025-12-04T14:26:42.1911369Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1911411Z     method(*args, **kwargs)
2025-12-04T14:26:42.1911560Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1911602Z     method(*args, **kwargs)
2025-12-04T14:26:42.1911752Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1911790Z     with policy():
2025-12-04T14:26:42.1911942Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1911986Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1912335Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 2. CUDA driver allocated memory was 2300575744 and is now 3837788160.
2025-12-04T14:26:42.1912340Z 
2025-12-04T14:26:42.1912413Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1912647Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1912649Z 
2025-12-04T14:26:42.1912735Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1912800Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1912862Z ====================== 1 failed, 19 deselected in 39.10s =======================
2025-12-04T14:26:42.1912900Z Got exit code 1
2025-12-04T14:26:42.1913079Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda
2025-12-04T14:26:42.1913208Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.1913395Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4c46e115b9b03122.xml
2025-12-04T14:26:42.1913454Z ============================= test session starts ==============================
2025-12-04T14:26:42.1913566Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1913611Z cachedir: .pytest_cache
2025-12-04T14:26:42.1913769Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1913817Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1913857Z configfile: pytest.ini
2025-12-04T14:26:42.1914020Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1914094Z collecting ... collected 60 items / 4 deselected / 56 selected
2025-12-04T14:26:42.1914148Z stepcurrent: skipping 4 already run items.
2025-12-04T14:26:42.1914190Z Running 16 items in this shard
2025-12-04T14:26:42.1914194Z 
2025-12-04T14:26:42.1914517Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda I1204 14:08:31.900000 377072 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 377141
2025-12-04T14:26:42.1914674Z I1204 14:08:31.901000 377072 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 377142
2025-12-04T14:26:42.1914835Z I1204 14:08:31.902000 377072 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 377143
2025-12-04T14:26:42.1914986Z I1204 14:08:31.902000 377072 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 377144
2025-12-04T14:26:42.1915576Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1915616Z   _warn_cpu_init()
2025-12-04T14:26:42.1916180Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1916220Z   _warn_cpu_init()
2025-12-04T14:26:42.1916827Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1916864Z   _warn_cpu_init()
2025-12-04T14:26:42.1917429Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1917469Z   _warn_cpu_init()
2025-12-04T14:26:42.1917761Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1917807Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1917949Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1918114Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1918403Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1918560Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1918845Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1919007Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1919288Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1919450Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1919744Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1919891Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1920173Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1920310Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1920592Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1920743Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1921222Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3965714432.
2025-12-04T14:26:42.1921340Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1921535Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1921891Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1922006Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1922222Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1922389Z [rank0]:E1204 14:09:09.282000 377141 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1922429Z dist init r=0, world=4
2025-12-04T14:26:42.1922568Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1922726Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1923016Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1923188Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1923477Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1923616Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1923894Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1924054Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1924330Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1924478Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1924756Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1924895Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1925172Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1925323Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1925796Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 2. CUDA driver allocated memory was 2300575744 and is now 3812622336.
2025-12-04T14:26:42.1925910Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1926107Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1926462Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1926578Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1926823Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1926990Z [rank2]:E1204 14:09:09.286000 377143 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1927031Z dist init r=2, world=4
2025-12-04T14:26:42.1927169Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1927329Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1927641Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1927809Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1928092Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1928233Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1928509Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1928659Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1928937Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1929083Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1929361Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1929497Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1929777Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1929925Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1930395Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3762290688.
2025-12-04T14:26:42.1930511Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1930707Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1931059Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1931172Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1931384Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1931547Z [rank3]:E1204 14:09:09.287000 377144 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1931587Z dist init r=3, world=4
2025-12-04T14:26:42.1931751Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1931910Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1932208Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1932371Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1932655Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1932780Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1933056Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1933204Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1933482Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1933631Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1933907Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1934046Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1934323Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1934473Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1934948Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3829399552.
2025-12-04T14:26:42.1935063Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1935260Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1935612Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1935728Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1935939Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1936124Z [rank1]:E1204 14:09:09.292000 377142 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1936173Z dist init r=1, world=4
2025-12-04T14:26:42.1936508Z [rank0]:[W1204 14:09:09.971394531 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1936561Z FAILED [39.2428s] [  6%]
2025-12-04T14:26:42.1936563Z 
2025-12-04T14:26:42.1936618Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1936719Z _____ TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda _____
2025-12-04T14:26:42.1936812Z Traceback (most recent call last):
2025-12-04T14:26:42.1936979Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1937022Z     self._join_processes(fn)
2025-12-04T14:26:42.1937195Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1937248Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1937427Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1937470Z     raise RuntimeError(error)
2025-12-04T14:26:42.1937551Z RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1937595Z Traceback (most recent call last):
2025-12-04T14:26:42.1937758Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1937798Z     getattr(self, test_name)()
2025-12-04T14:26:42.1937958Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1937992Z     fn()
2025-12-04T14:26:42.1938146Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1938186Z     method(*args, **kwargs)
2025-12-04T14:26:42.1938337Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1938376Z     method(*args, **kwargs)
2025-12-04T14:26:42.1938528Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1938565Z     with policy():
2025-12-04T14:26:42.1938717Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1938759Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1939107Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 2. CUDA driver allocated memory was 2300575744 and is now 3812622336.
2025-12-04T14:26:42.1939112Z 
2025-12-04T14:26:42.1939188Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1939415Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1939418Z 
2025-12-04T14:26:42.1939508Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1939511Z 
2025-12-04T14:26:42.1939512Z 
2025-12-04T14:26:42.1939587Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1939675Z Process 2 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1939942Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4c46e115b9b03122.xml -
2025-12-04T14:26:42.1940019Z =========================== short test summary info ============================
2025-12-04T14:26:42.1940262Z FAILED [39.2428s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda - RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.1940308Z Traceback (most recent call last):
2025-12-04T14:26:42.1940495Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1940538Z     getattr(self, test_name)()
2025-12-04T14:26:42.1940697Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1940732Z     fn()
2025-12-04T14:26:42.1940884Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1940923Z     method(*args, **kwargs)
2025-12-04T14:26:42.1941077Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1941117Z     method(*args, **kwargs)
2025-12-04T14:26:42.1941267Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1941307Z     with policy():
2025-12-04T14:26:42.1941459Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1941499Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1941850Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 2. CUDA driver allocated memory was 2300575744 and is now 3812622336.
2025-12-04T14:26:42.1941853Z 
2025-12-04T14:26:42.1941927Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1942153Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1942156Z 
2025-12-04T14:26:42.1942243Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1942306Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1942368Z ======================= 1 failed, 4 deselected in 39.40s =======================
2025-12-04T14:26:42.1942407Z Got exit code 1
2025-12-04T14:26:42.1942448Z Retrying single test...
2025-12-04T14:26:42.1942639Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d1ef7a7263a660cc.xml
2025-12-04T14:26:42.1942702Z ============================= test session starts ==============================
2025-12-04T14:26:42.1942813Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1942857Z cachedir: .pytest_cache
2025-12-04T14:26:42.1943015Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1943062Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1943103Z configfile: pytest.ini
2025-12-04T14:26:42.1943266Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1943340Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1943560Z stepcurrent: skipping 4 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1943624Z Running 1 items in this shard
2025-12-04T14:26:42.1943626Z 
2025-12-04T14:26:42.1943928Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda I1204 14:09:13.529000 377474 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 377543
2025-12-04T14:26:42.1944096Z I1204 14:09:13.530000 377474 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 377544
2025-12-04T14:26:42.1944260Z I1204 14:09:13.530000 377474 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 377545
2025-12-04T14:26:42.1944411Z I1204 14:09:13.531000 377474 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 377546
2025-12-04T14:26:42.1944990Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1945031Z   _warn_cpu_init()
2025-12-04T14:26:42.1945596Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1945634Z   _warn_cpu_init()
2025-12-04T14:26:42.1946199Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1946236Z   _warn_cpu_init()
2025-12-04T14:26:42.1947047Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1947086Z   _warn_cpu_init()
2025-12-04T14:26:42.1947378Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1947423Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1947564Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1947730Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1948020Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1948176Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1948490Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1948628Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1948906Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1949066Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1949346Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1949492Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1949771Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1949907Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1950190Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1950339Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1950813Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3965714432.
2025-12-04T14:26:42.1950931Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1951127Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1951483Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1951596Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1951809Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1951973Z [rank0]:E1204 14:09:50.963000 377543 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1952013Z dist init r=0, world=4
2025-12-04T14:26:42.1952152Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1952311Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1952617Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1952771Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1953066Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1953203Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1953479Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1953629Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1960134Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1960294Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1960579Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1960716Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1960998Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1961146Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1961621Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 2. CUDA driver allocated memory was 2300575744 and is now 3812622336.
2025-12-04T14:26:42.1961739Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1961937Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1962293Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1962406Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1962620Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1962786Z [rank2]:E1204 14:09:50.968000 377545 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.1962826Z dist init r=2, world=4
2025-12-04T14:26:42.1963019Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1963182Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1963487Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1963641Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1963944Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1964069Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1964349Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1964497Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1964772Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1964919Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1965195Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1965330Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1965609Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1965756Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1966229Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3762290688.
2025-12-04T14:26:42.1966347Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1966542Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1966937Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1967052Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1967262Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1967451Z [rank3]:E1204 14:09:50.985000 377546 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1967490Z dist init r=3, world=4
2025-12-04T14:26:42.1967643Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1967801Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1968090Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1968258Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1968544Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1968669Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1968944Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1969093Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1969368Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1969517Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1969792Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1969927Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1970208Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1970355Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1970832Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3829399552.
2025-12-04T14:26:42.1970947Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1971141Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1971494Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1971628Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1971841Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1972016Z [rank1]:E1204 14:09:51.030000 377544 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1972056Z dist init r=1, world=4
2025-12-04T14:26:42.1972392Z [rank0]:[W1204 14:09:51.631225156 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.1972447Z FAILED [39.2413s] [100%]
2025-12-04T14:26:42.1972449Z 
2025-12-04T14:26:42.1972508Z =================================== FAILURES ===================================
2025-12-04T14:26:42.1972609Z _____ TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda _____
2025-12-04T14:26:42.1972656Z Traceback (most recent call last):
2025-12-04T14:26:42.1972820Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.1972864Z     self._join_processes(fn)
2025-12-04T14:26:42.1973037Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.1973093Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.1973269Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.1973313Z     raise RuntimeError(error)
2025-12-04T14:26:42.1973393Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1973439Z Traceback (most recent call last):
2025-12-04T14:26:42.1973600Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1973643Z     getattr(self, test_name)()
2025-12-04T14:26:42.1973801Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1973837Z     fn()
2025-12-04T14:26:42.1973987Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1974030Z     method(*args, **kwargs)
2025-12-04T14:26:42.1974181Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1974223Z     method(*args, **kwargs)
2025-12-04T14:26:42.1974371Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1974409Z     with policy():
2025-12-04T14:26:42.1974560Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1974602Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1974956Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3965714432.
2025-12-04T14:26:42.1974958Z 
2025-12-04T14:26:42.1975037Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1975267Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1975270Z 
2025-12-04T14:26:42.1975358Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1975360Z 
2025-12-04T14:26:42.1975362Z 
2025-12-04T14:26:42.1975460Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.1975548Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.1975795Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d1ef7a7263a660cc.xml -
2025-12-04T14:26:42.1975855Z =========================== short test summary info ============================
2025-12-04T14:26:42.1976118Z FAILED [39.2413s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.1976166Z Traceback (most recent call last):
2025-12-04T14:26:42.1976330Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1976373Z     getattr(self, test_name)()
2025-12-04T14:26:42.1976533Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1976570Z     fn()
2025-12-04T14:26:42.1976720Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1976798Z     method(*args, **kwargs)
2025-12-04T14:26:42.1976948Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1976990Z     method(*args, **kwargs)
2025-12-04T14:26:42.1977139Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1977177Z     with policy():
2025-12-04T14:26:42.1977328Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1977368Z     raise RuntimeError(msg)
2025-12-04T14:26:42.1977717Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3965714432.
2025-12-04T14:26:42.1977721Z 
2025-12-04T14:26:42.1977796Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1978022Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1978027Z 
2025-12-04T14:26:42.1978112Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1978175Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.1978237Z ====================== 1 failed, 19 deselected in 39.41s =======================
2025-12-04T14:26:42.1978275Z Got exit code 1
2025-12-04T14:26:42.1978315Z Retrying single test...
2025-12-04T14:26:42.1978504Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-5b5eb822e5f1b7dd.xml
2025-12-04T14:26:42.1978562Z ============================= test session starts ==============================
2025-12-04T14:26:42.1978676Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.1978717Z cachedir: .pytest_cache
2025-12-04T14:26:42.1978876Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.1978923Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.1978963Z configfile: pytest.ini
2025-12-04T14:26:42.1979127Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.1979228Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.1979449Z stepcurrent: skipping 4 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1979506Z Running 1 items in this shard
2025-12-04T14:26:42.1979508Z 
2025-12-04T14:26:42.1979807Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda I1204 14:09:55.268000 377876 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 377945
2025-12-04T14:26:42.1979980Z I1204 14:09:55.269000 377876 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 377946
2025-12-04T14:26:42.1980133Z I1204 14:09:55.270000 377876 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 377947
2025-12-04T14:26:42.1980283Z I1204 14:09:55.270000 377876 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 377948
2025-12-04T14:26:42.1980865Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1980904Z   _warn_cpu_init()
2025-12-04T14:26:42.1981473Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1981511Z   _warn_cpu_init()
2025-12-04T14:26:42.1982078Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1982115Z   _warn_cpu_init()
2025-12-04T14:26:42.1982678Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.1982717Z   _warn_cpu_init()
2025-12-04T14:26:42.1983008Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.1983052Z   return func(*args, **kwargs)
2025-12-04T14:26:42.1983197Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1983358Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1983668Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1983823Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1984123Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1984259Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1984537Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1984686Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1984962Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1985110Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1985383Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1985522Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1985800Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1985949Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1986433Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3965714432.
2025-12-04T14:26:42.1986550Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1986790Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1987143Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1987259Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1987472Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1987639Z [rank0]:E1204 14:10:32.558000 377945 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.1987677Z dist init r=0, world=4
2025-12-04T14:26:42.1987815Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1987999Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1988287Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1988458Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1988760Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1988884Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1989161Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1989310Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1989586Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1989733Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1990010Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1990147Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1990426Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1990573Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1991049Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3829399552.
2025-12-04T14:26:42.1991166Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1991360Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1991713Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1991826Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1992042Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1992226Z [rank1]:E1204 14:10:32.559000 377946 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.1992266Z dist init r=1, world=4
2025-12-04T14:26:42.1992403Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1992579Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1992866Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1993032Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1993320Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1993444Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1993723Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1993871Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1994150Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1994299Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1994574Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1994713Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1994990Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1995139Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.1995616Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 3. CUDA driver allocated memory was 2250244096 and is now 3762290688.
2025-12-04T14:26:42.1995732Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1995929Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.1996281Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.1996395Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.1996628Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.1996841Z [rank3]:E1204 14:10:32.596000 377948 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.1996879Z dist init r=3, world=4
2025-12-04T14:26:42.1997016Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.1997189Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.1997476Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.1997634Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.1997920Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.1998043Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.1998318Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1998466Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1998741Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.1998888Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.1999163Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.1999299Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.1999577Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.1999725Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2000200Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 2. CUDA driver allocated memory was 2300575744 and is now 3812622336.
2025-12-04T14:26:42.2000316Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2000511Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2000885Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.2001014Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2001225Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2001398Z [rank2]:E1204 14:10:32.605000 377947 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2001436Z dist init r=2, world=4
2025-12-04T14:26:42.2001772Z [rank0]:[W1204 14:10:32.228566118 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2001815Z FAILED [39.2414s] [100%]
2025-12-04T14:26:42.2001817Z 
2025-12-04T14:26:42.2001871Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2001971Z _____ TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda _____
2025-12-04T14:26:42.2002018Z Traceback (most recent call last):
2025-12-04T14:26:42.2002180Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2002225Z     self._join_processes(fn)
2025-12-04T14:26:42.2002399Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2002454Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2002630Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2002676Z     raise RuntimeError(error)
2025-12-04T14:26:42.2002755Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2002803Z Traceback (most recent call last):
2025-12-04T14:26:42.2002962Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2003006Z     getattr(self, test_name)()
2025-12-04T14:26:42.2003164Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2003201Z     fn()
2025-12-04T14:26:42.2003352Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2003394Z     method(*args, **kwargs)
2025-12-04T14:26:42.2003543Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2003585Z     method(*args, **kwargs)
2025-12-04T14:26:42.2003733Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2003772Z     with policy():
2025-12-04T14:26:42.2003922Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2003964Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2004311Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3965714432.
2025-12-04T14:26:42.2004316Z 
2025-12-04T14:26:42.2004389Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2004639Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.2004641Z 
2025-12-04T14:26:42.2004729Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2004740Z 
2025-12-04T14:26:42.2004801Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2004846Z Traceback (most recent call last):
2025-12-04T14:26:42.2005009Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2005060Z     getattr(self, test_name)()
2025-12-04T14:26:42.2005218Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2005252Z     fn()
2025-12-04T14:26:42.2005401Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2005440Z     method(*args, **kwargs)
2025-12-04T14:26:42.2005591Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2005629Z     method(*args, **kwargs)
2025-12-04T14:26:42.2005779Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2005816Z     with policy():
2025-12-04T14:26:42.2005967Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2006008Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2006353Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3829399552.
2025-12-04T14:26:42.2006356Z 
2025-12-04T14:26:42.2006429Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2006656Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.2006659Z 
2025-12-04T14:26:42.2006784Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2006786Z 
2025-12-04T14:26:42.2006788Z 
2025-12-04T14:26:42.2006863Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2006952Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2007186Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-5b5eb822e5f1b7dd.xml -
2025-12-04T14:26:42.2007246Z =========================== short test summary info ============================
2025-12-04T14:26:42.2007492Z FAILED [39.2414s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2007538Z Traceback (most recent call last):
2025-12-04T14:26:42.2007703Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2007745Z     getattr(self, test_name)()
2025-12-04T14:26:42.2007904Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2008810Z     fn()
2025-12-04T14:26:42.2008961Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2008999Z     method(*args, **kwargs)
2025-12-04T14:26:42.2009151Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2009189Z     method(*args, **kwargs)
2025-12-04T14:26:42.2009379Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2009429Z     with policy():
2025-12-04T14:26:42.2009584Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2009625Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2009973Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 0. CUDA driver allocated memory was 2453667840 and is now 3965714432.
2025-12-04T14:26:42.2009991Z 
2025-12-04T14:26:42.2010065Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2010295Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.2010297Z 
2025-12-04T14:26:42.2010385Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2010388Z 
2025-12-04T14:26:42.2010446Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2010493Z Traceback (most recent call last):
2025-12-04T14:26:42.2010654Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2010698Z     getattr(self, test_name)()
2025-12-04T14:26:42.2010855Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2010889Z     fn()
2025-12-04T14:26:42.2011039Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2011078Z     method(*args, **kwargs)
2025-12-04T14:26:42.2011234Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2011274Z     method(*args, **kwargs)
2025-12-04T14:26:42.2011423Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2011461Z     with policy():
2025-12-04T14:26:42.2011610Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2011653Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2011996Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3829399552.
2025-12-04T14:26:42.2011998Z 
2025-12-04T14:26:42.2012071Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2012298Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.2012304Z 
2025-12-04T14:26:42.2012391Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2012454Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2012517Z ====================== 1 failed, 19 deselected in 39.40s =======================
2025-12-04T14:26:42.2012556Z Got exit code 1
2025-12-04T14:26:42.2012733Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda
2025-12-04T14:26:42.2012861Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2013071Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-baff277b2331bb4f.xml
2025-12-04T14:26:42.2013131Z ============================= test session starts ==============================
2025-12-04T14:26:42.2013257Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2013298Z cachedir: .pytest_cache
2025-12-04T14:26:42.2013455Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2013501Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2013554Z configfile: pytest.ini
2025-12-04T14:26:42.2013714Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2013787Z collecting ... collected 60 items / 5 deselected / 55 selected
2025-12-04T14:26:42.2013840Z stepcurrent: skipping 5 already run items.
2025-12-04T14:26:42.2013883Z Running 15 items in this shard
2025-12-04T14:26:42.2013885Z 
2025-12-04T14:26:42.2014203Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda I1204 14:10:36.900000 378278 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 378347
2025-12-04T14:26:42.2014357Z I1204 14:10:36.901000 378278 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 378348
2025-12-04T14:26:42.2014510Z I1204 14:10:36.902000 378278 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 378349
2025-12-04T14:26:42.2014662Z I1204 14:10:36.902000 378278 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 378350
2025-12-04T14:26:42.2014955Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2015006Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2015580Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2015620Z   _warn_cpu_init()
2025-12-04T14:26:42.2015908Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2015957Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2016525Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2016564Z   _warn_cpu_init()
2025-12-04T14:26:42.2016892Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2016972Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2017288Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2017363Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2017647Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2017710Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2018282Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2018331Z   _warn_cpu_init()
2025-12-04T14:26:42.2018618Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2018693Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2018976Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2019025Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2019593Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2019632Z   _warn_cpu_init()
2025-12-04T14:26:42.2019914Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2019988Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2021273Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2021401Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2021630Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2021674Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2022962Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2023107Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2023334Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2023376Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2024633Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2024757Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2024979Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2025021Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2026276Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2026398Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2026643Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2026684Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2026955Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2027009Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2027230Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2027283Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2027503Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2027542Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2027763Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2027802Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2028092Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2028133Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2028277Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2028441Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2028732Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2028888Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2029174Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2029299Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2029577Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2029726Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2030004Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2030152Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2030427Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2030564Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2030870Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2031018Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2031520Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 152064 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2031648Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2031843Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2032212Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2032326Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2032538Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2032704Z [rank1]:E1204 14:10:44.268000 378348 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2032743Z dist init r=1, world=4
2025-12-04T14:26:42.2032880Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2033040Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2033328Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2033481Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2033767Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2033890Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2034172Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2034321Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2034596Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2034743Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2035016Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2035171Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2035446Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2035606Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2036111Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 166400 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2036226Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2036423Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2036830Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2036945Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2037155Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2037322Z [rank2]:E1204 14:10:44.282000 378349 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2037360Z dist init r=2, world=4
2025-12-04T14:26:42.2037498Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2037659Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2037945Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2038101Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2038387Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2038512Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2038792Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2038941Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2039216Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2039389Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2039665Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2039815Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2040092Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2040252Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2040738Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 160256 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2040855Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2041050Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2041415Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2041528Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2041739Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2041902Z [rank0]:E1204 14:10:44.287000 378347 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2041940Z dist init r=0, world=4
2025-12-04T14:26:42.2042076Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2042237Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2042523Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2042676Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2042962Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2043085Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2043366Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2043512Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2043809Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2043966Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2044241Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2044387Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2044665Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2044814Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2045298Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 160256 on device 3. CUDA driver allocated memory was 2250244096 and is now 3808428032.
2025-12-04T14:26:42.2045413Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2045608Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2045974Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2046087Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2046297Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2046462Z [rank3]:E1204 14:10:44.326000 378350 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2046499Z dist init r=3, world=4
2025-12-04T14:26:42.2046888Z [rank0]:[W1204 14:10:44.015359240 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2046928Z FAILED [9.3152s] [  6%]
2025-12-04T14:26:42.2046930Z 
2025-12-04T14:26:42.2046987Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2047092Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2047138Z Traceback (most recent call last):
2025-12-04T14:26:42.2047303Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2047346Z     self._join_processes(fn)
2025-12-04T14:26:42.2047518Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2047571Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2047772Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2047816Z     raise RuntimeError(error)
2025-12-04T14:26:42.2047897Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2047956Z Traceback (most recent call last):
2025-12-04T14:26:42.2048118Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2048160Z     getattr(self, test_name)()
2025-12-04T14:26:42.2048317Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2048368Z     fn()
2025-12-04T14:26:42.2048520Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2048559Z     method(*args, **kwargs)
2025-12-04T14:26:42.2048710Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2048751Z     method(*args, **kwargs)
2025-12-04T14:26:42.2048901Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2048938Z     with policy():
2025-12-04T14:26:42.2049090Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2049129Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2049490Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 160256 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2049493Z 
2025-12-04T14:26:42.2049569Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2049808Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2049811Z 
2025-12-04T14:26:42.2049899Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2049901Z 
2025-12-04T14:26:42.2049961Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2050006Z Traceback (most recent call last):
2025-12-04T14:26:42.2050168Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2050211Z     getattr(self, test_name)()
2025-12-04T14:26:42.2050368Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2050404Z     fn()
2025-12-04T14:26:42.2050551Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2050593Z     method(*args, **kwargs)
2025-12-04T14:26:42.2050740Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2050781Z     method(*args, **kwargs)
2025-12-04T14:26:42.2050931Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2050967Z     with policy():
2025-12-04T14:26:42.2051117Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2051158Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2051518Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 152064 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2051521Z 
2025-12-04T14:26:42.2051616Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2051855Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2051875Z 
2025-12-04T14:26:42.2051961Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2051963Z 
2025-12-04T14:26:42.2051965Z 
2025-12-04T14:26:42.2052040Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2052140Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2052374Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-baff277b2331bb4f.xml -
2025-12-04T14:26:42.2052434Z =========================== short test summary info ============================
2025-12-04T14:26:42.2052693Z FAILED [9.3152s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2052741Z Traceback (most recent call last):
2025-12-04T14:26:42.2052903Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2052946Z     getattr(self, test_name)()
2025-12-04T14:26:42.2053105Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2053140Z     fn()
2025-12-04T14:26:42.2053289Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2053329Z     method(*args, **kwargs)
2025-12-04T14:26:42.2053480Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2053519Z     method(*args, **kwargs)
2025-12-04T14:26:42.2053667Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2053707Z     with policy():
2025-12-04T14:26:42.2053857Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2053897Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2054256Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 160256 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2054260Z 
2025-12-04T14:26:42.2054331Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2054568Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2054572Z 
2025-12-04T14:26:42.2054656Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2054658Z 
2025-12-04T14:26:42.2054719Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2054763Z Traceback (most recent call last):
2025-12-04T14:26:42.2054926Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2054968Z     getattr(self, test_name)()
2025-12-04T14:26:42.2055125Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2055158Z     fn()
2025-12-04T14:26:42.2055307Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2055369Z     method(*args, **kwargs)
2025-12-04T14:26:42.2055520Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2055570Z     method(*args, **kwargs)
2025-12-04T14:26:42.2055718Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2055754Z     with policy():
2025-12-04T14:26:42.2055903Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2055956Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2056312Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 152064 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2056315Z 
2025-12-04T14:26:42.2056388Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2056623Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2056626Z 
2025-12-04T14:26:42.2056713Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2056809Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2056873Z ======================= 1 failed, 5 deselected in 9.48s ========================
2025-12-04T14:26:42.2056910Z Got exit code 1
2025-12-04T14:26:42.2056951Z Retrying single test...
2025-12-04T14:26:42.2057142Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1473018dfba133d5.xml
2025-12-04T14:26:42.2057201Z ============================= test session starts ==============================
2025-12-04T14:26:42.2057313Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2057356Z cachedir: .pytest_cache
2025-12-04T14:26:42.2057514Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2057561Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2057600Z configfile: pytest.ini
2025-12-04T14:26:42.2057762Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2057837Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2058068Z stepcurrent: skipping 5 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2058112Z Running 1 items in this shard
2025-12-04T14:26:42.2058116Z 
2025-12-04T14:26:42.2058430Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda I1204 14:10:48.695000 378680 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 378749
2025-12-04T14:26:42.2058585Z I1204 14:10:48.696000 378680 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 378750
2025-12-04T14:26:42.2058736Z I1204 14:10:48.696000 378680 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 378751
2025-12-04T14:26:42.2058887Z I1204 14:10:48.697000 378680 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 378752
2025-12-04T14:26:42.2059178Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2059257Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2059838Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2059903Z   _warn_cpu_init()
2025-12-04T14:26:42.2060189Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2060238Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2060807Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2060843Z   _warn_cpu_init()
2025-12-04T14:26:42.2061129Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2061209Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2061493Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2061568Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2061852Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2061901Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2062182Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2062231Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2062798Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2062836Z   _warn_cpu_init()
2025-12-04T14:26:42.2063402Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2063439Z   _warn_cpu_init()
2025-12-04T14:26:42.2063745Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2063818Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2064115Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2064187Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2065478Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2065606Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2065835Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2065877Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2067194Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2067322Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2067547Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2067591Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2068886Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2069030Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2070289Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2070424Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2070650Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2070693Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2070916Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2070957Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2071176Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2071217Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2071435Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2071477Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2071696Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2071736Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2071957Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2071997Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2072287Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2072326Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2072472Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2072635Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2072948Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2073104Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2073401Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2073524Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2073813Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2073964Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2074242Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2074391Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2074665Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2074802Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2075080Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2075228Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2075717Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 168448 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2075832Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2076030Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2076397Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2076511Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2076726Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2076930Z [rank0]:E1204 14:10:56.189000 378749 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2076973Z dist init r=0, world=4
2025-12-04T14:26:42.2077112Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2077302Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2077604Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2077760Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2078056Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2078182Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2078463Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2078611Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2078889Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2079037Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2079315Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2079452Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2079732Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2079880Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2080368Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 172544 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2080487Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2080687Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2081057Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2081171Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2081386Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2081571Z [rank1]:E1204 14:10:56.196000 378750 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2081611Z dist init r=1, world=4
2025-12-04T14:26:42.2081764Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2081923Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2082212Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2082378Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2082666Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2082789Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2083071Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2083223Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2083498Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2083647Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2083922Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2084061Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2084337Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2084487Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2084975Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 166400 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2085089Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2085286Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2085652Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2085794Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2086006Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2086183Z [rank2]:E1204 14:10:56.204000 378751 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2086221Z dist init r=2, world=4
2025-12-04T14:26:42.2086357Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2086528Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2086853Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2087010Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2087295Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2087419Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2087699Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2087847Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2088127Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2088276Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2088551Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2088689Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2088970Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2089118Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2089603Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 168448 on device 3. CUDA driver allocated memory was 2250244096 and is now 3808428032.
2025-12-04T14:26:42.2089719Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2089915Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2090311Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2090438Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2090651Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2090828Z [rank3]:E1204 14:10:56.207000 378752 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2090869Z dist init r=3, world=4
2025-12-04T14:26:42.2091206Z [rank0]:[W1204 14:10:56.943136683 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2091246Z FAILED [9.4165s] [100%]
2025-12-04T14:26:42.2091249Z 
2025-12-04T14:26:42.2091308Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2091411Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2091460Z Traceback (most recent call last):
2025-12-04T14:26:42.2091623Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2091670Z     self._join_processes(fn)
2025-12-04T14:26:42.2091842Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2091896Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2092076Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2092120Z     raise RuntimeError(error)
2025-12-04T14:26:42.2092202Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2092249Z Traceback (most recent call last):
2025-12-04T14:26:42.2092409Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2092454Z     getattr(self, test_name)()
2025-12-04T14:26:42.2092613Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2092649Z     fn()
2025-12-04T14:26:42.2092801Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2092846Z     method(*args, **kwargs)
2025-12-04T14:26:42.2092999Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2093043Z     method(*args, **kwargs)
2025-12-04T14:26:42.2093193Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2093234Z     with policy():
2025-12-04T14:26:42.2093387Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2093428Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2093791Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 168448 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2093793Z 
2025-12-04T14:26:42.2093868Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2094130Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2094143Z 
2025-12-04T14:26:42.2094232Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2094234Z 
2025-12-04T14:26:42.2094236Z 
2025-12-04T14:26:42.2094313Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2094401Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2094648Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1473018dfba133d5.xml -
2025-12-04T14:26:42.2094710Z =========================== short test summary info ============================
2025-12-04T14:26:42.2094969Z FAILED [9.4165s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2095017Z Traceback (most recent call last):
2025-12-04T14:26:42.2095180Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2095226Z     getattr(self, test_name)()
2025-12-04T14:26:42.2095388Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2095425Z     fn()
2025-12-04T14:26:42.2095575Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2095617Z     method(*args, **kwargs)
2025-12-04T14:26:42.2095765Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2095805Z     method(*args, **kwargs)
2025-12-04T14:26:42.2095955Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2095993Z     with policy():
2025-12-04T14:26:42.2096145Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2096188Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2096548Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 168448 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2096553Z 
2025-12-04T14:26:42.2096627Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2096904Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2096907Z 
2025-12-04T14:26:42.2096995Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2097059Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2097121Z ======================= 1 failed, 19 deselected in 9.56s =======================
2025-12-04T14:26:42.2097160Z Got exit code 1
2025-12-04T14:26:42.2097199Z Retrying single test...
2025-12-04T14:26:42.2097391Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-59afc4c2d3eac168.xml
2025-12-04T14:26:42.2097450Z ============================= test session starts ==============================
2025-12-04T14:26:42.2097563Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2097604Z cachedir: .pytest_cache
2025-12-04T14:26:42.2097795Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2097842Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2097884Z configfile: pytest.ini
2025-12-04T14:26:42.2098060Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2098135Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2098365Z stepcurrent: skipping 5 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2098432Z Running 1 items in this shard
2025-12-04T14:26:42.2098434Z 
2025-12-04T14:26:42.2098751Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda I1204 14:11:00.715000 379082 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 379151
2025-12-04T14:26:42.2098907Z I1204 14:11:00.715000 379082 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 379152
2025-12-04T14:26:42.2099061Z I1204 14:11:00.716000 379082 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 379153
2025-12-04T14:26:42.2099212Z I1204 14:11:00.717000 379082 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 379154
2025-12-04T14:26:42.2099506Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2099558Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2099849Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2099898Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2100477Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2100518Z   _warn_cpu_init()
2025-12-04T14:26:42.2101084Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2101124Z   _warn_cpu_init()
2025-12-04T14:26:42.2101410Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2101489Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2101776Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2101852Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2102159Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2102209Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2102789Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2102842Z   _warn_cpu_init()
2025-12-04T14:26:42.2103128Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2103203Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2103495Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2103546Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2104111Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2104150Z   _warn_cpu_init()
2025-12-04T14:26:42.2104435Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2104509Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2105781Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2105908Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2106138Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2106182Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2107507Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2107661Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2107888Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2107932Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2109192Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2109316Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2109544Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2109588Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2110853Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2110977Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2111205Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2111246Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2111507Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2111549Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2111770Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2111822Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2112043Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2112096Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2112319Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2112359Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2112652Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2112695Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2112839Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2113004Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2113296Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2113453Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2113741Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2113869Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2114147Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2114300Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2114580Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2114728Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2115005Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2115142Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2115422Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2115569Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2116077Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 164352 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2116209Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2116405Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2116838Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2116956Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2117171Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2117336Z [rank2]:E1204 14:11:07.991000 379153 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2117377Z dist init r=2, world=4
2025-12-04T14:26:42.2117517Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2117677Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2117969Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2118124Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2118410Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2118532Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2118814Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2118962Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2119246Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2119397Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2119672Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2119812Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2120115Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2120265Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2120765Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 152064 on device 3. CUDA driver allocated memory was 2250244096 and is now 3808428032.
2025-12-04T14:26:42.2120894Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2121090Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2121456Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2121571Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2121782Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2121948Z [rank3]:E1204 14:11:07.997000 379154 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2121986Z dist init r=3, world=4
2025-12-04T14:26:42.2122125Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2122290Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2122576Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2122734Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2123020Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2123147Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2123424Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2123575Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2123852Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2124002Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2124278Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2124436Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2124726Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2124873Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2125368Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 168448 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2125486Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2125681Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2126048Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2126162Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2126379Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2126544Z [rank0]:E1204 14:11:08.008000 379151 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2126585Z dist init r=0, world=4
2025-12-04T14:26:42.2126723Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2126929Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2127217Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2127371Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2127657Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2127781Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2128057Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2128206Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2128484Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2128662Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2128938Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2129089Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2129364Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2129527Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2130011Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 152064 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2130126Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2130324Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2130688Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2130802Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2131011Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2131176Z [rank1]:E1204 14:11:08.010000 379152 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2131215Z dist init r=1, world=4
2025-12-04T14:26:42.2131553Z [rank0]:[W1204 14:11:08.700281461 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2131593Z FAILED [9.2148s] [100%]
2025-12-04T14:26:42.2131598Z 
2025-12-04T14:26:42.2131655Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2131760Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2131807Z Traceback (most recent call last):
2025-12-04T14:26:42.2131971Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2132016Z     self._join_processes(fn)
2025-12-04T14:26:42.2132191Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2132244Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2132423Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2132465Z     raise RuntimeError(error)
2025-12-04T14:26:42.2132546Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2132590Z Traceback (most recent call last):
2025-12-04T14:26:42.2132779Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2132820Z     getattr(self, test_name)()
2025-12-04T14:26:42.2132988Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2133022Z     fn()
2025-12-04T14:26:42.2133174Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2133231Z     method(*args, **kwargs)
2025-12-04T14:26:42.2133382Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2133422Z     method(*args, **kwargs)
2025-12-04T14:26:42.2133572Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2133609Z     with policy():
2025-12-04T14:26:42.2133765Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2133806Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2134170Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 152064 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2134173Z 
2025-12-04T14:26:42.2134249Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2134485Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2134487Z 
2025-12-04T14:26:42.2134576Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2134579Z 
2025-12-04T14:26:42.2134580Z 
2025-12-04T14:26:42.2134654Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2134745Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2134979Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-59afc4c2d3eac168.xml -
2025-12-04T14:26:42.2135043Z =========================== short test summary info ============================
2025-12-04T14:26:42.2135301Z FAILED [9.2148s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2135348Z Traceback (most recent call last):
2025-12-04T14:26:42.2135512Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2135553Z     getattr(self, test_name)()
2025-12-04T14:26:42.2135714Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2135750Z     fn()
2025-12-04T14:26:42.2135903Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2135941Z     method(*args, **kwargs)
2025-12-04T14:26:42.2136094Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2136134Z     method(*args, **kwargs)
2025-12-04T14:26:42.2136285Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2136322Z     with policy():
2025-12-04T14:26:42.2136475Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2136537Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2136939Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 152064 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2136957Z 
2025-12-04T14:26:42.2137030Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2137267Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2137285Z 
2025-12-04T14:26:42.2137374Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2137438Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2137500Z ======================= 1 failed, 19 deselected in 9.37s =======================
2025-12-04T14:26:42.2137537Z Got exit code 1
2025-12-04T14:26:42.2137724Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda
2025-12-04T14:26:42.2137852Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2138042Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-251bbc18c025af9f.xml
2025-12-04T14:26:42.2138101Z ============================= test session starts ==============================
2025-12-04T14:26:42.2138213Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2138253Z cachedir: .pytest_cache
2025-12-04T14:26:42.2138413Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2138461Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2138502Z configfile: pytest.ini
2025-12-04T14:26:42.2138662Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2138738Z collecting ... collected 60 items / 6 deselected / 54 selected
2025-12-04T14:26:42.2138790Z stepcurrent: skipping 6 already run items.
2025-12-04T14:26:42.2138835Z Running 14 items in this shard
2025-12-04T14:26:42.2138837Z 
2025-12-04T14:26:42.2139148Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda I1204 14:11:12.200000 379484 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 379553
2025-12-04T14:26:42.2139304Z I1204 14:11:12.200000 379484 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 379554
2025-12-04T14:26:42.2139460Z I1204 14:11:12.201000 379484 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 379555
2025-12-04T14:26:42.2139610Z I1204 14:11:12.202000 379484 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 379556
2025-12-04T14:26:42.2140193Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2140232Z   _warn_cpu_init()
2025-12-04T14:26:42.2140829Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2140879Z   _warn_cpu_init()
2025-12-04T14:26:42.2141440Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2141492Z   _warn_cpu_init()
2025-12-04T14:26:42.2142060Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2142099Z   _warn_cpu_init()
2025-12-04T14:26:42.2142389Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2142431Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2142574Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2142737Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2143031Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2143189Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2143477Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2143603Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2143884Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2144032Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2144311Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2144459Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2144736Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2144874Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2145174Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2145333Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2145819Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2145947Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2146144Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2146505Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2146620Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2146863Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2147028Z [rank1]:E1204 14:12:07.882000 379554 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2147068Z dist init r=1, world=4
2025-12-04T14:26:42.2147207Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2147370Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2147656Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2147812Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2148096Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2148223Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2148498Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2148649Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2148925Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2149070Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2149378Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2149534Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2149812Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2149974Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2150459Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2150573Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2150770Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2151130Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2151243Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2151456Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2151619Z [rank3]:E1204 14:12:07.892000 379556 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2151659Z dist init r=3, world=4
2025-12-04T14:26:42.2151797Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2151955Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2152243Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2152397Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2152684Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2152808Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2153085Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2153232Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2153530Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2153679Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2153964Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2154102Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2154390Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2154541Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2155022Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2155138Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2155334Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2155694Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2155807Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2156019Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2156183Z [rank0]:E1204 14:12:07.908000 379553 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2156222Z dist init r=0, world=4
2025-12-04T14:26:42.2156360Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2156520Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2156849Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2157006Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2157296Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2157420Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2157696Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2157870Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2158146Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2158307Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2158596Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2158731Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2159012Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2159161Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2159640Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2159757Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2159952Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2160312Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2160426Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2160639Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2160802Z [rank2]:E1204 14:12:07.919000 379555 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2160842Z dist init r=2, world=4
2025-12-04T14:26:42.2161181Z [rank0]:[W1204 14:12:08.773235623 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2161224Z FAILED [57.5578s] [  7%]
2025-12-04T14:26:42.2161226Z 
2025-12-04T14:26:42.2161284Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2161383Z __ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda ___
2025-12-04T14:26:42.2161431Z Traceback (most recent call last):
2025-12-04T14:26:42.2161593Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2161638Z     self._join_processes(fn)
2025-12-04T14:26:42.2161810Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2161884Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2162061Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2162117Z     raise RuntimeError(error)
2025-12-04T14:26:42.2162196Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2162242Z Traceback (most recent call last):
2025-12-04T14:26:42.2162402Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2162463Z     getattr(self, test_name)()
2025-12-04T14:26:42.2162619Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2162658Z     fn()
2025-12-04T14:26:42.2162807Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2162850Z     method(*args, **kwargs)
2025-12-04T14:26:42.2162999Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2163041Z     method(*args, **kwargs)
2025-12-04T14:26:42.2163190Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2163228Z     with policy():
2025-12-04T14:26:42.2163378Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2163420Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2163778Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2163780Z 
2025-12-04T14:26:42.2163856Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2164089Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2164092Z 
2025-12-04T14:26:42.2164178Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2164180Z 
2025-12-04T14:26:42.2164182Z 
2025-12-04T14:26:42.2164257Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2164346Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2164579Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-251bbc18c025af9f.xml -
2025-12-04T14:26:42.2164638Z =========================== short test summary info ============================
2025-12-04T14:26:42.2164891Z FAILED [57.5578s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2164941Z Traceback (most recent call last):
2025-12-04T14:26:42.2165104Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2165149Z     getattr(self, test_name)()
2025-12-04T14:26:42.2165306Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2165343Z     fn()
2025-12-04T14:26:42.2165493Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2165533Z     method(*args, **kwargs)
2025-12-04T14:26:42.2165683Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2165744Z     method(*args, **kwargs)
2025-12-04T14:26:42.2165896Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2165946Z     with policy():
2025-12-04T14:26:42.2166095Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2166137Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2166488Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2166501Z 
2025-12-04T14:26:42.2166576Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2166854Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2166856Z 
2025-12-04T14:26:42.2166943Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2167006Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2167067Z ======================= 1 failed, 6 deselected in 57.72s =======================
2025-12-04T14:26:42.2167104Z Got exit code 1
2025-12-04T14:26:42.2167144Z Retrying single test...
2025-12-04T14:26:42.2167336Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-e58cab743defc658.xml
2025-12-04T14:26:42.2167391Z ============================= test session starts ==============================
2025-12-04T14:26:42.2167503Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2167543Z cachedir: .pytest_cache
2025-12-04T14:26:42.2167702Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2167748Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2167790Z configfile: pytest.ini
2025-12-04T14:26:42.2167951Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2168027Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2168254Z stepcurrent: skipping 6 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2168299Z Running 1 items in this shard
2025-12-04T14:26:42.2168301Z 
2025-12-04T14:26:42.2168610Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda I1204 14:12:12.562000 379886 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 379955
2025-12-04T14:26:42.2168765Z I1204 14:12:12.563000 379886 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 379956
2025-12-04T14:26:42.2168918Z I1204 14:12:12.564000 379886 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 379957
2025-12-04T14:26:42.2169067Z I1204 14:12:12.564000 379886 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 379958
2025-12-04T14:26:42.2169673Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2169711Z   _warn_cpu_init()
2025-12-04T14:26:42.2170274Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2170346Z   _warn_cpu_init()
2025-12-04T14:26:42.2170912Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2170950Z   _warn_cpu_init()
2025-12-04T14:26:42.2171512Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2171550Z   _warn_cpu_init()
2025-12-04T14:26:42.2171841Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2171883Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2172027Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2172189Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2172476Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2172631Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2172920Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2173045Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2173325Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2173476Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2173754Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2173900Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2174201Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2174348Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2174624Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2174786Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2175296Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2175412Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2175608Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2175969Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2176088Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2176300Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2176465Z [rank2]:E1204 14:13:08.113000 379957 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2176506Z dist init r=2, world=4
2025-12-04T14:26:42.2176643Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2176847Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2177135Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2177291Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2177578Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2177704Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2177981Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2178130Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2178432Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2178579Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2178875Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2179011Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2179304Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2179453Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2179934Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2180050Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2180246Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2180606Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2180718Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2180931Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2181094Z [rank3]:E1204 14:13:08.140000 379958 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2181134Z dist init r=3, world=4
2025-12-04T14:26:42.2181273Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2181432Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2181720Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2181872Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2182158Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2182283Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2182560Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2182726Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2183013Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2183159Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2183444Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2183581Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2183858Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2184008Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2184486Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2184602Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2184799Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2185157Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2185271Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2185483Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2185647Z [rank1]:E1204 14:13:08.166000 379956 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2185684Z dist init r=1, world=4
2025-12-04T14:26:42.2185825Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2185984Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2186272Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2186428Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2186711Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2186906Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2187183Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2187345Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2187620Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2187782Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2188060Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2188197Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2188475Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2188624Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2189105Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2189221Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2189418Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2189778Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2189891Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2190104Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2190266Z [rank0]:E1204 14:13:08.198000 379955 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2190306Z dist init r=0, world=4
2025-12-04T14:26:42.2190642Z [rank0]:[W1204 14:13:08.073118276 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2190685Z FAILED [57.4580s] [100%]
2025-12-04T14:26:42.2190686Z 
2025-12-04T14:26:42.2190742Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2190841Z __ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda ___
2025-12-04T14:26:42.2190889Z Traceback (most recent call last):
2025-12-04T14:26:42.2191071Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2191116Z     self._join_processes(fn)
2025-12-04T14:26:42.2191288Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2191354Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2191533Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2191589Z     raise RuntimeError(error)
2025-12-04T14:26:42.2191669Z RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2191714Z Traceback (most recent call last):
2025-12-04T14:26:42.2191876Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2191919Z     getattr(self, test_name)()
2025-12-04T14:26:42.2192079Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2192114Z     fn()
2025-12-04T14:26:42.2192265Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2192306Z     method(*args, **kwargs)
2025-12-04T14:26:42.2192455Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2192496Z     method(*args, **kwargs)
2025-12-04T14:26:42.2192645Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2192685Z     with policy():
2025-12-04T14:26:42.2192835Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2192878Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2193234Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2193237Z 
2025-12-04T14:26:42.2193310Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2193543Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2193547Z 
2025-12-04T14:26:42.2193633Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2193635Z 
2025-12-04T14:26:42.2193696Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2193740Z Traceback (most recent call last):
2025-12-04T14:26:42.2193909Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2193950Z     getattr(self, test_name)()
2025-12-04T14:26:42.2194109Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2194144Z     fn()
2025-12-04T14:26:42.2194296Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2194335Z     method(*args, **kwargs)
2025-12-04T14:26:42.2194486Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2194525Z     method(*args, **kwargs)
2025-12-04T14:26:42.2194675Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2194712Z     with policy():
2025-12-04T14:26:42.2194889Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2201857Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2202240Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2202300Z 
2025-12-04T14:26:42.2202383Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2202644Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2202646Z 
2025-12-04T14:26:42.2202739Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2202741Z 
2025-12-04T14:26:42.2202743Z 
2025-12-04T14:26:42.2202829Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2202920Z Process 2 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2203164Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-e58cab743defc658.xml -
2025-12-04T14:26:42.2203228Z =========================== short test summary info ============================
2025-12-04T14:26:42.2203486Z FAILED [57.4580s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda - RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2203538Z Traceback (most recent call last):
2025-12-04T14:26:42.2203708Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2203755Z     getattr(self, test_name)()
2025-12-04T14:26:42.2203920Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2203958Z     fn()
2025-12-04T14:26:42.2204114Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2204158Z     method(*args, **kwargs)
2025-12-04T14:26:42.2204310Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2204354Z     method(*args, **kwargs)
2025-12-04T14:26:42.2204505Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2204543Z     with policy():
2025-12-04T14:26:42.2204698Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2204741Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2205106Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2205111Z 
2025-12-04T14:26:42.2205189Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2205434Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2205440Z 
2025-12-04T14:26:42.2205529Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2205531Z 
2025-12-04T14:26:42.2205592Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2205637Z Traceback (most recent call last):
2025-12-04T14:26:42.2205834Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2205877Z     getattr(self, test_name)()
2025-12-04T14:26:42.2206038Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2206084Z     fn()
2025-12-04T14:26:42.2206235Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2206275Z     method(*args, **kwargs)
2025-12-04T14:26:42.2206442Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2206486Z     method(*args, **kwargs)
2025-12-04T14:26:42.2206638Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2206675Z     with policy():
2025-12-04T14:26:42.2206875Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2206916Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2207273Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2207277Z 
2025-12-04T14:26:42.2207353Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2207587Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2207589Z 
2025-12-04T14:26:42.2207677Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2207741Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2207808Z ====================== 1 failed, 19 deselected in 57.62s =======================
2025-12-04T14:26:42.2207845Z Got exit code 1
2025-12-04T14:26:42.2207888Z Retrying single test...
2025-12-04T14:26:42.2208078Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-42fda556a7d871e9.xml
2025-12-04T14:26:42.2208138Z ============================= test session starts ==============================
2025-12-04T14:26:42.2208252Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2208295Z cachedir: .pytest_cache
2025-12-04T14:26:42.2208457Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2208504Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2208544Z configfile: pytest.ini
2025-12-04T14:26:42.2208714Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2208789Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2209020Z stepcurrent: skipping 6 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2209066Z Running 1 items in this shard
2025-12-04T14:26:42.2209068Z 
2025-12-04T14:26:42.2209381Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda I1204 14:13:12.517000 380288 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 380357
2025-12-04T14:26:42.2209540Z I1204 14:13:12.518000 380288 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 380358
2025-12-04T14:26:42.2209728Z I1204 14:13:12.518000 380288 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 380359
2025-12-04T14:26:42.2209882Z I1204 14:13:12.519000 380288 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 380360
2025-12-04T14:26:42.2210479Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2210533Z   _warn_cpu_init()
2025-12-04T14:26:42.2211101Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2211138Z   _warn_cpu_init()
2025-12-04T14:26:42.2211703Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2211740Z   _warn_cpu_init()
2025-12-04T14:26:42.2212313Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2212351Z   _warn_cpu_init()
2025-12-04T14:26:42.2212641Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2212687Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2212832Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2212996Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2213290Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2213449Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2213737Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2213865Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2214143Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2214315Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2214607Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2214756Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2215047Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2215185Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2215468Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2215619Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2216140Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2218882Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2219093Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2219460Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2219577Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2219792Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2219959Z [rank1]:E1204 14:14:08.132000 380358 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2219997Z dist init r=1, world=4
2025-12-04T14:26:42.2220139Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2220298Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2220589Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2220743Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2221028Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2221199Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2221476Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2221639Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2221916Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2222080Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2222359Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2222497Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2222774Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2222924Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2223408Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2223523Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2223720Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2224111Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2224233Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2224449Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2224613Z [rank2]:E1204 14:14:08.138000 380359 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2224653Z dist init r=2, world=4
2025-12-04T14:26:42.2224789Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2224949Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2225237Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2225391Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2225698Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2225833Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2226109Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2226266Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2226544Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2226691Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2227008Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2227142Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2227420Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2227569Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2228049Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2228163Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2228358Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2228719Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2228831Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2229045Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2229209Z [rank3]:E1204 14:14:08.140000 380360 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2229247Z dist init r=3, world=4
2025-12-04T14:26:42.2229385Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2229542Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2229856Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2230021Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2230305Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2230446Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2230723Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2230872Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2231148Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2231296Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2231572Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2231707Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2231984Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2232132Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2232610Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2232725Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2232921Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2233278Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2233391Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2233602Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2233767Z [rank0]:E1204 14:14:08.193000 380357 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2233805Z dist init r=0, world=4
2025-12-04T14:26:42.2234166Z [rank0]:[W1204 14:14:08.942131061 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2234217Z FAILED [57.5589s] [100%]
2025-12-04T14:26:42.2234219Z 
2025-12-04T14:26:42.2234275Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2234377Z __ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda ___
2025-12-04T14:26:42.2234422Z Traceback (most recent call last):
2025-12-04T14:26:42.2234604Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2234646Z     self._join_processes(fn)
2025-12-04T14:26:42.2234818Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2234871Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2235050Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2235094Z     raise RuntimeError(error)
2025-12-04T14:26:42.2235174Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2235218Z Traceback (most recent call last):
2025-12-04T14:26:42.2235379Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2235422Z     getattr(self, test_name)()
2025-12-04T14:26:42.2235580Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2235613Z     fn()
2025-12-04T14:26:42.2235766Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2235811Z     method(*args, **kwargs)
2025-12-04T14:26:42.2235962Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2236004Z     method(*args, **kwargs)
2025-12-04T14:26:42.2236155Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2236194Z     with policy():
2025-12-04T14:26:42.2236344Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2236386Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2236780Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2236783Z 
2025-12-04T14:26:42.2236862Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2237095Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2237101Z 
2025-12-04T14:26:42.2237188Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2237191Z 
2025-12-04T14:26:42.2237251Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2237295Z Traceback (most recent call last):
2025-12-04T14:26:42.2237461Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2237501Z     getattr(self, test_name)()
2025-12-04T14:26:42.2237660Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2237693Z     fn()
2025-12-04T14:26:42.2237874Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2237914Z     method(*args, **kwargs)
2025-12-04T14:26:42.2238079Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2238118Z     method(*args, **kwargs)
2025-12-04T14:26:42.2238268Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2238320Z     with policy():
2025-12-04T14:26:42.2238472Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2238511Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2238866Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2238869Z 
2025-12-04T14:26:42.2238942Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2239174Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2239176Z 
2025-12-04T14:26:42.2239264Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2239267Z 
2025-12-04T14:26:42.2239269Z 
2025-12-04T14:26:42.2239344Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2239433Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2239676Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-42fda556a7d871e9.xml -
2025-12-04T14:26:42.2239745Z =========================== short test summary info ============================
2025-12-04T14:26:42.2240006Z FAILED [57.5589s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2240053Z Traceback (most recent call last):
2025-12-04T14:26:42.2240214Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2240257Z     getattr(self, test_name)()
2025-12-04T14:26:42.2240418Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2240456Z     fn()
2025-12-04T14:26:42.2240608Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2240646Z     method(*args, **kwargs)
2025-12-04T14:26:42.2240801Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2240840Z     method(*args, **kwargs)
2025-12-04T14:26:42.2240989Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2241025Z     with policy():
2025-12-04T14:26:42.2241176Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2241216Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2241571Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2241573Z 
2025-12-04T14:26:42.2241671Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2241904Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2241918Z 
2025-12-04T14:26:42.2242003Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2242006Z 
2025-12-04T14:26:42.2242064Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2242110Z Traceback (most recent call last):
2025-12-04T14:26:42.2242283Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2242324Z     getattr(self, test_name)()
2025-12-04T14:26:42.2242482Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2242516Z     fn()
2025-12-04T14:26:42.2242667Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2242707Z     method(*args, **kwargs)
2025-12-04T14:26:42.2242857Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2242896Z     method(*args, **kwargs)
2025-12-04T14:26:42.2243047Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2243084Z     with policy():
2025-12-04T14:26:42.2243235Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2243275Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2243627Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2243630Z 
2025-12-04T14:26:42.2243703Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2243933Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2243936Z 
2025-12-04T14:26:42.2244022Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2244087Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2244150Z ====================== 1 failed, 19 deselected in 57.72s =======================
2025-12-04T14:26:42.2244187Z Got exit code 1
2025-12-04T14:26:42.2244368Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda
2025-12-04T14:26:42.2244498Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2244688Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-9d3e964f6fc57cf6.xml
2025-12-04T14:26:42.2244748Z ============================= test session starts ==============================
2025-12-04T14:26:42.2244861Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2244905Z cachedir: .pytest_cache
2025-12-04T14:26:42.2245064Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2245111Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2245151Z configfile: pytest.ini
2025-12-04T14:26:42.2245315Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2245409Z collecting ... collected 60 items / 7 deselected / 53 selected
2025-12-04T14:26:42.2245464Z stepcurrent: skipping 7 already run items.
2025-12-04T14:26:42.2245516Z Running 13 items in this shard
2025-12-04T14:26:42.2245518Z 
2025-12-04T14:26:42.2245843Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda I1204 14:14:12.374000 380690 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 380759
2025-12-04T14:26:42.2246012Z I1204 14:14:12.375000 380690 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 380760
2025-12-04T14:26:42.2246164Z I1204 14:14:12.375000 380690 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 380761
2025-12-04T14:26:42.2246316Z I1204 14:14:12.376000 380690 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 380762
2025-12-04T14:26:42.2246942Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2246982Z   _warn_cpu_init()
2025-12-04T14:26:42.2247551Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2247590Z   _warn_cpu_init()
2025-12-04T14:26:42.2248157Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2248194Z   _warn_cpu_init()
2025-12-04T14:26:42.2248766Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2248803Z   _warn_cpu_init()
2025-12-04T14:26:42.2249093Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2249137Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2249282Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2249446Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2249735Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2249921Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2250228Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2250353Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2250646Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2250795Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2251073Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2251221Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2251498Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2251635Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2251915Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2252064Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2252560Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2252677Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2252871Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2253249Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2253363Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2253579Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2253744Z [rank3]:E1204 14:15:08.023000 380762 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2253781Z dist init r=3, world=4
2025-12-04T14:26:42.2253921Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2254101Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2254388Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2254553Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2254840Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2254978Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2255257Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2255405Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2255682Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2255830Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2256105Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2256243Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2256519Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2256670Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2257213Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2257329Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2257523Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2257895Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2258009Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2258220Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2258412Z [rank1]:E1204 14:15:08.026000 380760 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2258451Z dist init r=1, world=4
2025-12-04T14:26:42.2258588Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2258761Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2259047Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2259218Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2259506Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2259633Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2259911Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2260059Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2260337Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2260483Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2260762Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2260897Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2261175Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2261323Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2261816Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2261931Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2262125Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2262498Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2262611Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2262844Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2263017Z [rank2]:E1204 14:15:08.071000 380761 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2263058Z dist init r=2, world=4
2025-12-04T14:26:42.2263194Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2263367Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2263655Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2263808Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2264096Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2264220Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2264498Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2264645Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2264924Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2265072Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2265346Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2265486Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2265764Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2265914Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2266407Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2266522Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2266718Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2267163Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2267291Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2267501Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2267681Z [rank0]:E1204 14:15:08.074000 380759 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2267718Z dist init r=0, world=4
2025-12-04T14:26:42.2268056Z [rank0]:[W1204 14:15:08.819318481 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2268098Z FAILED [57.5553s] [  7%]
2025-12-04T14:26:42.2268100Z 
2025-12-04T14:26:42.2268155Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2268270Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda _
2025-12-04T14:26:42.2268316Z Traceback (most recent call last):
2025-12-04T14:26:42.2268478Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2268523Z     self._join_processes(fn)
2025-12-04T14:26:42.2268698Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2268752Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2268930Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2268975Z     raise RuntimeError(error)
2025-12-04T14:26:42.2269057Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2269102Z Traceback (most recent call last):
2025-12-04T14:26:42.2269263Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2269306Z     getattr(self, test_name)()
2025-12-04T14:26:42.2269463Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2269497Z     fn()
2025-12-04T14:26:42.2269648Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2269687Z     method(*args, **kwargs)
2025-12-04T14:26:42.2269838Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2269878Z     method(*args, **kwargs)
2025-12-04T14:26:42.2270028Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2270065Z     with policy():
2025-12-04T14:26:42.2270217Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2270259Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2270622Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2270625Z 
2025-12-04T14:26:42.2270700Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2270970Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2270972Z 
2025-12-04T14:26:42.2271077Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2271079Z 
2025-12-04T14:26:42.2271138Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2271185Z Traceback (most recent call last):
2025-12-04T14:26:42.2271345Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2271399Z     getattr(self, test_name)()
2025-12-04T14:26:42.2271556Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2271591Z     fn()
2025-12-04T14:26:42.2271739Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2271782Z     method(*args, **kwargs)
2025-12-04T14:26:42.2271932Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2271972Z     method(*args, **kwargs)
2025-12-04T14:26:42.2272120Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2272157Z     with policy():
2025-12-04T14:26:42.2272306Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2272347Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2272708Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2272710Z 
2025-12-04T14:26:42.2272785Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2273028Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2273031Z 
2025-12-04T14:26:42.2273118Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2273120Z 
2025-12-04T14:26:42.2273122Z 
2025-12-04T14:26:42.2273198Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2273286Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2273523Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-9d3e964f6fc57cf6.xml -
2025-12-04T14:26:42.2273590Z =========================== short test summary info ============================
2025-12-04T14:26:42.2273854Z FAILED [57.5553s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2273900Z Traceback (most recent call last):
2025-12-04T14:26:42.2274063Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2274103Z     getattr(self, test_name)()
2025-12-04T14:26:42.2274264Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2274298Z     fn()
2025-12-04T14:26:42.2274449Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2274488Z     method(*args, **kwargs)
2025-12-04T14:26:42.2274661Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2274700Z     method(*args, **kwargs)
2025-12-04T14:26:42.2274850Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2274899Z     with policy():
2025-12-04T14:26:42.2275050Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2275090Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2275465Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2275467Z 
2025-12-04T14:26:42.2275539Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2275787Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2275790Z 
2025-12-04T14:26:42.2275877Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2275879Z 
2025-12-04T14:26:42.2275937Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2275984Z Traceback (most recent call last):
2025-12-04T14:26:42.2276145Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2276188Z     getattr(self, test_name)()
2025-12-04T14:26:42.2276345Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2276380Z     fn()
2025-12-04T14:26:42.2276529Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2276570Z     method(*args, **kwargs)
2025-12-04T14:26:42.2276719Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2276796Z     method(*args, **kwargs)
2025-12-04T14:26:42.2276945Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2276983Z     with policy():
2025-12-04T14:26:42.2277133Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2277175Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2277534Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2277540Z 
2025-12-04T14:26:42.2277612Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2277858Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2277861Z 
2025-12-04T14:26:42.2277947Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2278011Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2278073Z ======================= 1 failed, 7 deselected in 57.71s =======================
2025-12-04T14:26:42.2278110Z Got exit code 1
2025-12-04T14:26:42.2278149Z Retrying single test...
2025-12-04T14:26:42.2278338Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-3084d56ebcd57432.xml
2025-12-04T14:26:42.2278422Z ============================= test session starts ==============================
2025-12-04T14:26:42.2278536Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2278590Z cachedir: .pytest_cache
2025-12-04T14:26:42.2278749Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2278795Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2278835Z configfile: pytest.ini
2025-12-04T14:26:42.2279012Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2279087Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2279324Z stepcurrent: skipping 7 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2279369Z Running 1 items in this shard
2025-12-04T14:26:42.2279371Z 
2025-12-04T14:26:42.2279692Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda I1204 14:15:12.543000 381092 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 381161
2025-12-04T14:26:42.2279848Z I1204 14:15:12.544000 381092 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 381162
2025-12-04T14:26:42.2280003Z I1204 14:15:12.544000 381092 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 381163
2025-12-04T14:26:42.2280154Z I1204 14:15:12.545000 381092 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 381164
2025-12-04T14:26:42.2280738Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2280776Z   _warn_cpu_init()
2025-12-04T14:26:42.2281346Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2281385Z   _warn_cpu_init()
2025-12-04T14:26:42.2281952Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2281990Z   _warn_cpu_init()
2025-12-04T14:26:42.2282557Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2282594Z   _warn_cpu_init()
2025-12-04T14:26:42.2282918Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2282973Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2283116Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2283277Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2283581Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2283736Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2284025Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2284152Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2284431Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2284581Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2284859Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2285012Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2285289Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2285426Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2285705Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2285854Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2286348Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2286465Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2286663Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2287110Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2287226Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2287455Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2287620Z [rank0]:E1204 14:16:08.015000 381161 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2287676Z dist init r=0, world=4
2025-12-04T14:26:42.2287813Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2287973Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2288261Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2288418Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2288702Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2288829Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2289107Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2289255Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2289534Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2289682Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2289961Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2290096Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2290376Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2290526Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2291017Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2291133Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2291349Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2291722Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2291847Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2292071Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2292236Z [rank2]:E1204 14:16:08.020000 381163 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2292273Z dist init r=2, world=4
2025-12-04T14:26:42.2292413Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2292571Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2292861Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2293014Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2293300Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2293425Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2293703Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2293852Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2294128Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2294276Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2294555Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2294692Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2294969Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2295118Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2295630Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2295745Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2295952Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2296323Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2296454Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2296667Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2296903Z [rank3]:E1204 14:16:08.021000 381164 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2296943Z dist init r=3, world=4
2025-12-04T14:26:42.2297079Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2297239Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2297526Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2297681Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2297965Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2298092Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2298367Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2298516Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2298792Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2298941Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2299219Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2299353Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2299631Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2299779Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2300300Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2300432Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2300640Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2301012Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2301125Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2301339Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2301503Z [rank1]:E1204 14:16:08.024000 381162 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2301543Z dist init r=1, world=4
2025-12-04T14:26:42.2301879Z [rank0]:[W1204 14:16:08.666447486 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2301918Z FAILED [57.2562s] [100%]
2025-12-04T14:26:42.2301920Z 
2025-12-04T14:26:42.2301977Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2302088Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda _
2025-12-04T14:26:42.2302136Z Traceback (most recent call last):
2025-12-04T14:26:42.2302298Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2302342Z     self._join_processes(fn)
2025-12-04T14:26:42.2302513Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2302570Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2302748Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2302792Z     raise RuntimeError(error)
2025-12-04T14:26:42.2302872Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2302919Z Traceback (most recent call last):
2025-12-04T14:26:42.2303079Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2303124Z     getattr(self, test_name)()
2025-12-04T14:26:42.2303280Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2303315Z     fn()
2025-12-04T14:26:42.2303464Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2303507Z     method(*args, **kwargs)
2025-12-04T14:26:42.2303657Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2303697Z     method(*args, **kwargs)
2025-12-04T14:26:42.2303871Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2303908Z     with policy():
2025-12-04T14:26:42.2304060Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2304112Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2304479Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2304493Z 
2025-12-04T14:26:42.2304569Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2304816Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2304819Z 
2025-12-04T14:26:42.2304907Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2304909Z 
2025-12-04T14:26:42.2304912Z 
2025-12-04T14:26:42.2304987Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2305077Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2305311Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-3084d56ebcd57432.xml -
2025-12-04T14:26:42.2305373Z =========================== short test summary info ============================
2025-12-04T14:26:42.2305633Z FAILED [57.2562s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2305679Z Traceback (most recent call last):
2025-12-04T14:26:42.2305844Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2305886Z     getattr(self, test_name)()
2025-12-04T14:26:42.2306045Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2306081Z     fn()
2025-12-04T14:26:42.2306231Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2306272Z     method(*args, **kwargs)
2025-12-04T14:26:42.2306422Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2306462Z     method(*args, **kwargs)
2025-12-04T14:26:42.2306611Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2306648Z     with policy():
2025-12-04T14:26:42.2306846Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2306888Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2307251Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2307254Z 
2025-12-04T14:26:42.2307328Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2307572Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2307574Z 
2025-12-04T14:26:42.2307659Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2307747Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2307810Z ====================== 1 failed, 19 deselected in 57.42s =======================
2025-12-04T14:26:42.2307861Z Got exit code 1
2025-12-04T14:26:42.2307900Z Retrying single test...
2025-12-04T14:26:42.2308088Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-36e181b8747bd291.xml
2025-12-04T14:26:42.2308144Z ============================= test session starts ==============================
2025-12-04T14:26:42.2308279Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2308319Z cachedir: .pytest_cache
2025-12-04T14:26:42.2308477Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2308521Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2308564Z configfile: pytest.ini
2025-12-04T14:26:42.2308725Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2308801Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2309042Z stepcurrent: skipping 7 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2309085Z Running 1 items in this shard
2025-12-04T14:26:42.2309087Z 
2025-12-04T14:26:42.2309411Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda I1204 14:16:12.176000 381494 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 381563
2025-12-04T14:26:42.2309566Z I1204 14:16:12.177000 381494 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 381564
2025-12-04T14:26:42.2309724Z I1204 14:16:12.177000 381494 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 381565
2025-12-04T14:26:42.2309874Z I1204 14:16:12.178000 381494 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 381566
2025-12-04T14:26:42.2310455Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2310496Z   _warn_cpu_init()
2025-12-04T14:26:42.2311063Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2311101Z   _warn_cpu_init()
2025-12-04T14:26:42.2311663Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2311701Z   _warn_cpu_init()
2025-12-04T14:26:42.2312289Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2312343Z   _warn_cpu_init()
2025-12-04T14:26:42.2312634Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2312688Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2312830Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2312992Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2313282Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2313437Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2313724Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2313850Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2314129Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2314278Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2314554Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2314701Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2314976Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2315114Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2315397Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2315546Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2316038Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2316154Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2316375Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2316799Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2316931Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2317143Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2317306Z [rank0]:E1204 14:17:07.641000 381563 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2317346Z dist init r=0, world=4
2025-12-04T14:26:42.2317483Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2317645Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2317933Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2318088Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2318370Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2318497Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2318777Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2318924Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2319201Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2319347Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2319624Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2319760Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2320039Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2320189Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2320713Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2320842Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2321036Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2321420Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2321532Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2321744Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2321909Z [rank3]:E1204 14:17:07.648000 381566 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2321947Z dist init r=3, world=4
2025-12-04T14:26:42.2322086Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2322246Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2322536Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2322690Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2322975Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2323098Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2323375Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2323525Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2323801Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2323949Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2324223Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2324360Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2324640Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2324813Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2325303Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2325439Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2325634Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2326005Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2326119Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2326332Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2326495Z [rank2]:E1204 14:17:07.661000 381565 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2326536Z dist init r=2, world=4
2025-12-04T14:26:42.2326673Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2326877Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2327170Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2327325Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2327612Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2327738Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2328018Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2328165Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2328444Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2328590Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2328869Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2329005Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2329321Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2329485Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2329974Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2330100Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2330295Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2330668Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2330779Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2330991Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2331155Z [rank1]:E1204 14:17:07.727000 381564 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2331194Z dist init r=1, world=4
2025-12-04T14:26:42.2331534Z [rank0]:[W1204 14:17:07.296831843 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2331574Z FAILED [57.1573s] [100%]
2025-12-04T14:26:42.2331577Z 
2025-12-04T14:26:42.2331633Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2331744Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda _
2025-12-04T14:26:42.2331791Z Traceback (most recent call last):
2025-12-04T14:26:42.2331954Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2332001Z     self._join_processes(fn)
2025-12-04T14:26:42.2332175Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2332231Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2332413Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2332459Z     raise RuntimeError(error)
2025-12-04T14:26:42.2332541Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2332588Z Traceback (most recent call last):
2025-12-04T14:26:42.2332748Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2332793Z     getattr(self, test_name)()
2025-12-04T14:26:42.2332949Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2332986Z     fn()
2025-12-04T14:26:42.2333161Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2333201Z     method(*args, **kwargs)
2025-12-04T14:26:42.2333355Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2333408Z     method(*args, **kwargs)
2025-12-04T14:26:42.2333560Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2333597Z     with policy():
2025-12-04T14:26:42.2333763Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2333804Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2334175Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2334178Z 
2025-12-04T14:26:42.2334252Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2334501Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2334503Z 
2025-12-04T14:26:42.2334590Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2334595Z 
2025-12-04T14:26:42.2334597Z 
2025-12-04T14:26:42.2334672Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2334761Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2334992Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-36e181b8747bd291.xml -
2025-12-04T14:26:42.2335056Z =========================== short test summary info ============================
2025-12-04T14:26:42.2335317Z FAILED [57.1573s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2335367Z Traceback (most recent call last):
2025-12-04T14:26:42.2335530Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2335575Z     getattr(self, test_name)()
2025-12-04T14:26:42.2335735Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2335772Z     fn()
2025-12-04T14:26:42.2335922Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2335963Z     method(*args, **kwargs)
2025-12-04T14:26:42.2336114Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2336157Z     method(*args, **kwargs)
2025-12-04T14:26:42.2336306Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2336344Z     with policy():
2025-12-04T14:26:42.2336494Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2336538Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2336944Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2336946Z 
2025-12-04T14:26:42.2337047Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2337293Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2337310Z 
2025-12-04T14:26:42.2337396Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2337461Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2337537Z ====================== 1 failed, 19 deselected in 57.32s =======================
2025-12-04T14:26:42.2337576Z Got exit code 1
2025-12-04T14:26:42.2337769Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2337898Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2338085Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-49855475054bb596.xml
2025-12-04T14:26:42.2338145Z ============================= test session starts ==============================
2025-12-04T14:26:42.2338258Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2338301Z cachedir: .pytest_cache
2025-12-04T14:26:42.2338458Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2338507Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2338551Z configfile: pytest.ini
2025-12-04T14:26:42.2338713Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2338788Z collecting ... collected 60 items / 8 deselected / 52 selected
2025-12-04T14:26:42.2338841Z stepcurrent: skipping 8 already run items.
2025-12-04T14:26:42.2338886Z Running 12 items in this shard
2025-12-04T14:26:42.2338888Z 
2025-12-04T14:26:42.2339203Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda I1204 14:17:11.877000 381896 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 381965
2025-12-04T14:26:42.2339360Z I1204 14:17:11.877000 381896 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 381966
2025-12-04T14:26:42.2339512Z I1204 14:17:11.878000 381896 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 381967
2025-12-04T14:26:42.2339664Z I1204 14:17:11.879000 381896 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 381968
2025-12-04T14:26:42.2339957Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2340008Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2340583Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2340622Z   _warn_cpu_init()
2025-12-04T14:26:42.2340913Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2340990Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2341304Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2341363Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2341650Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2341713Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2342287Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2342327Z   _warn_cpu_init()
2025-12-04T14:26:42.2342614Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2342662Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2343232Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2343272Z   _warn_cpu_init()
2025-12-04T14:26:42.2343562Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2343611Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2344181Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2344218Z   _warn_cpu_init()
2025-12-04T14:26:42.2344508Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2344584Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2344871Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2344948Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2345234Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2345310Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2345572Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2345617Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2345852Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2345895Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2346117Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2346171Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2346391Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2346435Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2346655Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2346697Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2346948Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2346989Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2347209Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2347249Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2347469Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2347511Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2347657Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2347821Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2348113Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2348270Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2348556Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2348683Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2348961Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2349112Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2349389Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2349539Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2349845Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2350001Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2350278Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2350444Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2350935Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 103936 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2351052Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2351249Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2351614Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2351732Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2351947Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2352111Z [rank1]:E1204 14:17:19.485000 381966 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2352154Z dist init r=1, world=4
2025-12-04T14:26:42.2352293Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2352457Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2352745Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2352899Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2353183Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2353310Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2353588Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2353737Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2354039Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2354188Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2354475Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2354624Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2354905Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2355055Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2355539Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 101888 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2355654Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2355849Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2356214Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2356326Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2356541Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2356707Z [rank3]:E1204 14:17:19.486000 381968 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2356810Z dist init r=3, world=4
2025-12-04T14:26:42.2356950Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2357110Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2357398Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2357555Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2357840Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2357964Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2358273Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2358423Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2358712Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2358860Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2359153Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2359291Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2359569Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2359720Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2360204Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2360319Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2360518Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2360883Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2360999Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2361211Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2361378Z [rank2]:E1204 14:17:19.505000 381967 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2361418Z dist init r=2, world=4
2025-12-04T14:26:42.2361558Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2361723Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2362011Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2362167Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2362451Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2362599Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2362876Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2363038Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2363327Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2363474Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2363753Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2363892Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2364172Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2364322Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2364808Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 99840 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2364924Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2365119Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2365482Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2365596Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2365812Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2365976Z [rank0]:E1204 14:17:19.560000 381965 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2366017Z dist init r=0, world=4
2025-12-04T14:26:42.2366356Z [rank0]:[W1204 14:17:19.345503588 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2366397Z FAILED [9.4142s] [  8%]
2025-12-04T14:26:42.2366399Z 
2025-12-04T14:26:42.2366457Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2366556Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda _
2025-12-04T14:26:42.2366603Z Traceback (most recent call last):
2025-12-04T14:26:42.2366866Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2366911Z     self._join_processes(fn)
2025-12-04T14:26:42.2367097Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2367151Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2367327Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2367386Z     raise RuntimeError(error)
2025-12-04T14:26:42.2367467Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2367514Z Traceback (most recent call last):
2025-12-04T14:26:42.2367674Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2367718Z     getattr(self, test_name)()
2025-12-04T14:26:42.2367877Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2367916Z     fn()
2025-12-04T14:26:42.2368067Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2368109Z     method(*args, **kwargs)
2025-12-04T14:26:42.2368259Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2368301Z     method(*args, **kwargs)
2025-12-04T14:26:42.2368452Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2368487Z     with policy():
2025-12-04T14:26:42.2368643Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2368682Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2369044Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 103936 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2369047Z 
2025-12-04T14:26:42.2369121Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2369360Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2369363Z 
2025-12-04T14:26:42.2369449Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2369451Z 
2025-12-04T14:26:42.2369513Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2369557Z Traceback (most recent call last):
2025-12-04T14:26:42.2369721Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2369761Z     getattr(self, test_name)()
2025-12-04T14:26:42.2369920Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2369955Z     fn()
2025-12-04T14:26:42.2370102Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2370143Z     method(*args, **kwargs)
2025-12-04T14:26:42.2370292Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2370331Z     method(*args, **kwargs)
2025-12-04T14:26:42.2370478Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2370515Z     with policy():
2025-12-04T14:26:42.2370688Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2370731Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2371101Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 101888 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2371114Z 
2025-12-04T14:26:42.2371191Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2371427Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2371429Z 
2025-12-04T14:26:42.2371517Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2371519Z 
2025-12-04T14:26:42.2371521Z 
2025-12-04T14:26:42.2371600Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2371686Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2371921Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-49855475054bb596.xml -
2025-12-04T14:26:42.2371981Z =========================== short test summary info ============================
2025-12-04T14:26:42.2372239Z FAILED [9.4142s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2372285Z Traceback (most recent call last):
2025-12-04T14:26:42.2372451Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2372493Z     getattr(self, test_name)()
2025-12-04T14:26:42.2372656Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2372691Z     fn()
2025-12-04T14:26:42.2372845Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2372884Z     method(*args, **kwargs)
2025-12-04T14:26:42.2373037Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2373076Z     method(*args, **kwargs)
2025-12-04T14:26:42.2373228Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2373265Z     with policy():
2025-12-04T14:26:42.2373419Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2373461Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2373820Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 103936 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2373823Z 
2025-12-04T14:26:42.2373898Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2374134Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2374138Z 
2025-12-04T14:26:42.2374226Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2374228Z 
2025-12-04T14:26:42.2374287Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2374336Z Traceback (most recent call last):
2025-12-04T14:26:42.2374517Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2374560Z     getattr(self, test_name)()
2025-12-04T14:26:42.2374736Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2374774Z     fn()
2025-12-04T14:26:42.2374923Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2374975Z     method(*args, **kwargs)
2025-12-04T14:26:42.2375123Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2375165Z     method(*args, **kwargs)
2025-12-04T14:26:42.2375316Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2375351Z     with policy():
2025-12-04T14:26:42.2375502Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2375543Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2375903Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 101888 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2375906Z 
2025-12-04T14:26:42.2375979Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2376215Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2376218Z 
2025-12-04T14:26:42.2376303Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2376370Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2376432Z ======================= 1 failed, 8 deselected in 9.57s ========================
2025-12-04T14:26:42.2376475Z Got exit code 1
2025-12-04T14:26:42.2376516Z Retrying single test...
2025-12-04T14:26:42.2376706Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4780afafcc30b547.xml
2025-12-04T14:26:42.2376796Z ============================= test session starts ==============================
2025-12-04T14:26:42.2376909Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2376952Z cachedir: .pytest_cache
2025-12-04T14:26:42.2377108Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2377158Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2377198Z configfile: pytest.ini
2025-12-04T14:26:42.2377361Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2377436Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2377669Z stepcurrent: skipping 8 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2377713Z Running 1 items in this shard
2025-12-04T14:26:42.2377716Z 
2025-12-04T14:26:42.2378034Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda I1204 14:17:23.774000 382298 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 382367
2025-12-04T14:26:42.2378187Z I1204 14:17:23.775000 382298 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 382368
2025-12-04T14:26:42.2378373Z I1204 14:17:23.775000 382298 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 382369
2025-12-04T14:26:42.2378525Z I1204 14:17:23.776000 382298 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 382370
2025-12-04T14:26:42.2378829Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2378896Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2379474Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2379514Z   _warn_cpu_init()
2025-12-04T14:26:42.2379801Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2379880Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2380171Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2380215Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2380504Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2380553Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2381127Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2381164Z   _warn_cpu_init()
2025-12-04T14:26:42.2381449Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2381497Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2381781Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2381830Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2382397Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2382435Z   _warn_cpu_init()
2025-12-04T14:26:42.2383022Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2383071Z   _warn_cpu_init()
2025-12-04T14:26:42.2383359Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2383449Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2383736Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2383812Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2384096Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2384170Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2384398Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2384439Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2384665Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2384705Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2384928Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2384968Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2385195Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2385234Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2385453Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2385495Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2385713Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2385753Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2385971Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2386013Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2386230Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2386270Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2386413Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2386577Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2386946Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2387104Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2387405Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2387530Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2387823Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2387971Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2388250Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2388397Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2388674Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2388811Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2389091Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2389240Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2389730Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 112128 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2389852Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2390050Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2390417Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2390533Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2390744Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2390911Z [rank2]:E1204 14:17:31.248000 382369 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2390949Z dist init r=2, world=4
2025-12-04T14:26:42.2391088Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2391275Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2391563Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2391727Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2392025Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2392151Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2392429Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2392578Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2392854Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2393002Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2393276Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2393414Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2393691Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2393841Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2394327Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 116224 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2394443Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2394640Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2395005Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2395119Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2395332Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2395516Z [rank3]:E1204 14:17:31.253000 382370 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2395556Z dist init r=3, world=4
2025-12-04T14:26:42.2395694Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2395865Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2396152Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2396317Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2396603Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2396728Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2397043Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2397190Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2397467Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2397615Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2397891Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2398026Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2398304Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2398453Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2398936Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 116224 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2399051Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2399247Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2399611Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2399751Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2399964Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2400141Z [rank0]:E1204 14:17:31.325000 382367 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2400179Z dist init r=0, world=4
2025-12-04T14:26:42.2400317Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2400488Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2400777Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2400930Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2401218Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2401341Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2401621Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2401770Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2402047Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2402195Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2402471Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2402610Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2402886Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2403036Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2403519Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 103936 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2403632Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2403829Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2404213Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2404340Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2404550Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2404727Z [rank1]:E1204 14:17:31.326000 382368 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2404767Z dist init r=1, world=4
2025-12-04T14:26:42.2405102Z [rank0]:[W1204 14:17:31.159032598 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2405142Z FAILED [9.4154s] [100%]
2025-12-04T14:26:42.2405144Z 
2025-12-04T14:26:42.2405198Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2405301Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda _
2025-12-04T14:26:42.2405346Z Traceback (most recent call last):
2025-12-04T14:26:42.2405509Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2405554Z     self._join_processes(fn)
2025-12-04T14:26:42.2405726Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2405779Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2405962Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2406005Z     raise RuntimeError(error)
2025-12-04T14:26:42.2406086Z RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2406130Z Traceback (most recent call last):
2025-12-04T14:26:42.2406292Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2406333Z     getattr(self, test_name)()
2025-12-04T14:26:42.2406491Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2406525Z     fn()
2025-12-04T14:26:42.2406677Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2406719Z     method(*args, **kwargs)
2025-12-04T14:26:42.2406914Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2406956Z     method(*args, **kwargs)
2025-12-04T14:26:42.2407105Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2407144Z     with policy():
2025-12-04T14:26:42.2407294Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2407336Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2407694Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 112128 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2407697Z 
2025-12-04T14:26:42.2407773Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2408048Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2408050Z 
2025-12-04T14:26:42.2408150Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2408152Z 
2025-12-04T14:26:42.2408154Z 
2025-12-04T14:26:42.2408229Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2408316Z Process 2 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2408576Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4780afafcc30b547.xml -
2025-12-04T14:26:42.2408635Z =========================== short test summary info ============================
2025-12-04T14:26:42.2408894Z FAILED [9.4154s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda - RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2408939Z Traceback (most recent call last):
2025-12-04T14:26:42.2409103Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2409146Z     getattr(self, test_name)()
2025-12-04T14:26:42.2409305Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2409339Z     fn()
2025-12-04T14:26:42.2409490Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2409529Z     method(*args, **kwargs)
2025-12-04T14:26:42.2409680Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2409718Z     method(*args, **kwargs)
2025-12-04T14:26:42.2409871Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2409907Z     with policy():
2025-12-04T14:26:42.2410059Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2410099Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2410458Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 112128 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2410461Z 
2025-12-04T14:26:42.2410535Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2410773Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2410775Z 
2025-12-04T14:26:42.2410863Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2410924Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2410987Z ======================= 1 failed, 19 deselected in 9.58s =======================
2025-12-04T14:26:42.2411023Z Got exit code 1
2025-12-04T14:26:42.2411063Z Retrying single test...
2025-12-04T14:26:42.2411251Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-2189e6a6fccbb3b1.xml
2025-12-04T14:26:42.2411310Z ============================= test session starts ==============================
2025-12-04T14:26:42.2411420Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2411462Z cachedir: .pytest_cache
2025-12-04T14:26:42.2411618Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2411683Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2411723Z configfile: pytest.ini
2025-12-04T14:26:42.2411885Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2411970Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2412201Z stepcurrent: skipping 8 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2412258Z Running 1 items in this shard
2025-12-04T14:26:42.2412260Z 
2025-12-04T14:26:42.2412571Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda I1204 14:17:35.670000 382700 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 382769
2025-12-04T14:26:42.2412727Z I1204 14:17:35.671000 382700 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 382770
2025-12-04T14:26:42.2412879Z I1204 14:17:35.672000 382700 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 382771
2025-12-04T14:26:42.2413032Z I1204 14:17:35.672000 382700 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 382772
2025-12-04T14:26:42.2413322Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2413375Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2413952Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2413989Z   _warn_cpu_init()
2025-12-04T14:26:42.2414277Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2414326Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2414896Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2414932Z   _warn_cpu_init()
2025-12-04T14:26:42.2415219Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2415298Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2415584Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2415661Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2415972Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2416016Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2416298Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2416359Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2416972Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2417027Z   _warn_cpu_init()
2025-12-04T14:26:42.2417314Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2417361Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.2417931Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2417969Z   _warn_cpu_init()
2025-12-04T14:26:42.2418255Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2418329Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2418614Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2418687Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2418917Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2418959Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2419183Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2419225Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2419447Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2419491Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2419711Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2419754Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2419972Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2420013Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2420261Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2420302Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2420519Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2420573Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2420792Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2420848Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2420994Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2421156Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2421448Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2421603Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2421889Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2422014Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2422290Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2422442Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2422719Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2422869Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2423145Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2423284Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2423561Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2423711Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2424194Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 112128 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2424310Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2424526Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2424897Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2425024Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2425247Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2425411Z [rank3]:E1204 14:17:43.261000 382772 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2425451Z dist init r=3, world=4
2025-12-04T14:26:42.2425590Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2425749Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2426035Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2426191Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2426474Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2429008Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2429297Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2429448Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2429733Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2429882Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2430159Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2430296Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2430574Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2430723Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2431254Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2431371Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2431582Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2431949Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2432086Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2432297Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2432462Z [rank1]:E1204 14:17:43.262000 382770 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2432501Z dist init r=1, world=4
2025-12-04T14:26:42.2432639Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2432799Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2433087Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2433243Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2433530Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2433655Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2433931Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2434080Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2434359Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2434507Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2434785Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2434920Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2435198Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2435345Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2435853Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2435979Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2436186Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2436551Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2436666Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2436924Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2437087Z [rank0]:E1204 14:17:43.338000 382769 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2437126Z dist init r=0, world=4
2025-12-04T14:26:42.2437264Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2437424Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2437711Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2437864Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2438150Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2438273Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2438551Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2438698Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2438976Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2439125Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2439399Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2439536Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2439840Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2439988Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2440491Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2440622Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2440818Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2441181Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2441294Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2441505Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2441671Z [rank2]:E1204 14:17:43.339000 382771 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2441708Z dist init r=2, world=4
2025-12-04T14:26:42.2442045Z [rank0]:[W1204 14:17:43.249637154 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2442084Z FAILED [9.5137s] [100%]
2025-12-04T14:26:42.2442088Z 
2025-12-04T14:26:42.2442144Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2442247Z _ TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda _
2025-12-04T14:26:42.2442292Z Traceback (most recent call last):
2025-12-04T14:26:42.2442458Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2442500Z     self._join_processes(fn)
2025-12-04T14:26:42.2442673Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2442727Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2442906Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2442949Z     raise RuntimeError(error)
2025-12-04T14:26:42.2443029Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2443073Z Traceback (most recent call last):
2025-12-04T14:26:42.2443233Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2443275Z     getattr(self, test_name)()
2025-12-04T14:26:42.2443433Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2443467Z     fn()
2025-12-04T14:26:42.2443620Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2443659Z     method(*args, **kwargs)
2025-12-04T14:26:42.2443830Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2443870Z     method(*args, **kwargs)
2025-12-04T14:26:42.2444032Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2444068Z     with policy():
2025-12-04T14:26:42.2444219Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2444270Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2444629Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2444632Z 
2025-12-04T14:26:42.2444709Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2444948Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2444951Z 
2025-12-04T14:26:42.2445040Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2445042Z 
2025-12-04T14:26:42.2445101Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2445146Z Traceback (most recent call last):
2025-12-04T14:26:42.2445309Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2445351Z     getattr(self, test_name)()
2025-12-04T14:26:42.2445509Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2445544Z     fn()
2025-12-04T14:26:42.2445694Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2445734Z     method(*args, **kwargs)
2025-12-04T14:26:42.2445884Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2445923Z     method(*args, **kwargs)
2025-12-04T14:26:42.2446071Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2446112Z     with policy():
2025-12-04T14:26:42.2446264Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2446303Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2446661Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 112128 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2446664Z 
2025-12-04T14:26:42.2446737Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2447018Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2447021Z 
2025-12-04T14:26:42.2447107Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2447110Z 
2025-12-04T14:26:42.2447112Z 
2025-12-04T14:26:42.2447189Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2447277Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2447514Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-2189e6a6fccbb3b1.xml -
2025-12-04T14:26:42.2447606Z =========================== short test summary info ============================
2025-12-04T14:26:42.2447861Z FAILED [9.5137s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2447923Z Traceback (most recent call last):
2025-12-04T14:26:42.2448085Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2448143Z     getattr(self, test_name)()
2025-12-04T14:26:42.2448303Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2448338Z     fn()
2025-12-04T14:26:42.2448487Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2448530Z     method(*args, **kwargs)
2025-12-04T14:26:42.2448684Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2448723Z     method(*args, **kwargs)
2025-12-04T14:26:42.2448872Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2448908Z     with policy():
2025-12-04T14:26:42.2449057Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2449098Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2449453Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 95744 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2449457Z 
2025-12-04T14:26:42.2449530Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2449767Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2449771Z 
2025-12-04T14:26:42.2449855Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2449857Z 
2025-12-04T14:26:42.2449915Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2449959Z Traceback (most recent call last):
2025-12-04T14:26:42.2450121Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2450161Z     getattr(self, test_name)()
2025-12-04T14:26:42.2450318Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2450350Z     fn()
2025-12-04T14:26:42.2450502Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2450540Z     method(*args, **kwargs)
2025-12-04T14:26:42.2450690Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2450728Z     method(*args, **kwargs)
2025-12-04T14:26:42.2450876Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2450913Z     with policy():
2025-12-04T14:26:42.2451063Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2451102Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2451478Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 112128 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2451480Z 
2025-12-04T14:26:42.2451553Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2451797Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2451799Z 
2025-12-04T14:26:42.2451884Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2451959Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2452020Z ======================= 1 failed, 19 deselected in 9.68s =======================
2025-12-04T14:26:42.2452058Z Got exit code 1
2025-12-04T14:26:42.2452243Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda
2025-12-04T14:26:42.2452372Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2452564Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d8feba8dedb219c1.xml
2025-12-04T14:26:42.2452622Z ============================= test session starts ==============================
2025-12-04T14:26:42.2452736Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2452777Z cachedir: .pytest_cache
2025-12-04T14:26:42.2452936Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2452982Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2453021Z configfile: pytest.ini
2025-12-04T14:26:42.2453184Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2453257Z collecting ... collected 60 items / 9 deselected / 51 selected
2025-12-04T14:26:42.2453309Z stepcurrent: skipping 9 already run items.
2025-12-04T14:26:42.2453352Z Running 11 items in this shard
2025-12-04T14:26:42.2453354Z 
2025-12-04T14:26:42.2453660Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda I1204 14:17:47.729000 383102 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 383171
2025-12-04T14:26:42.2453815Z I1204 14:17:47.730000 383102 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 383172
2025-12-04T14:26:42.2453967Z I1204 14:17:47.731000 383102 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 383173
2025-12-04T14:26:42.2454117Z I1204 14:17:47.731000 383102 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 383174
2025-12-04T14:26:42.2454701Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2454742Z   _warn_cpu_init()
2025-12-04T14:26:42.2455044Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2455081Z   _init_core_state(
2025-12-04T14:26:42.2455605Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2455680Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2456248Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2456300Z   _warn_cpu_init()
2025-12-04T14:26:42.2456596Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2456633Z   _init_core_state(
2025-12-04T14:26:42.2457169Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2457231Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2457801Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2457838Z   _warn_cpu_init()
2025-12-04T14:26:42.2458131Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2458168Z   _init_core_state(
2025-12-04T14:26:42.2458655Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2458715Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2459282Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2459320Z   _warn_cpu_init()
2025-12-04T14:26:42.2459809Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2459866Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2460385Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2460453Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2460760Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2460802Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2461097Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2461133Z   _init_core_state(
2025-12-04T14:26:42.2461618Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2461676Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2462163Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2462221Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2462451Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2462493Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2462719Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2462759Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2462981Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2463021Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2463244Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2463284Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2463505Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2463543Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2463763Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2463803Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2464022Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2464060Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2464302Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2464351Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2464497Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2464661Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2464965Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2465122Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2465410Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2465536Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2465813Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2465964Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2466242Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2466390Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2466666Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2467044Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2467329Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2467480Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2467957Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 0. CUDA driver allocated memory was 2453667840 and is now 17620271104.
2025-12-04T14:26:42.2468075Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2468270Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2468626Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2468776Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2468991Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2469167Z [rank0]:E1204 14:17:56.390000 383171 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2469208Z dist init r=0, world=4
2025-12-04T14:26:42.2469365Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2469525Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2469817Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2469971Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2470259Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2470385Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2470663Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2470811Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2471090Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2471239Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2471515Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2471652Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2471933Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2472082Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2472558Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 1. CUDA driver allocated memory was 2317352960 and is now 17483956224.
2025-12-04T14:26:42.2472674Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2472872Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2473252Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2473377Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2473588Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2473765Z [rank1]:E1204 14:17:56.417000 383172 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2473803Z dist init r=1, world=4
2025-12-04T14:26:42.2473942Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2474103Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2474393Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2474548Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2474833Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2474958Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2475237Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2475386Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2475662Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2475811Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2476089Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2476226Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2476507Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2476656Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2477164Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 3. CUDA driver allocated memory was 2250244096 and is now 17416847360.
2025-12-04T14:26:42.2477311Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2477508Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2477875Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2478001Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2478213Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2478378Z [rank3]:E1204 14:17:56.432000 383174 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2478417Z dist init r=3, world=4
2025-12-04T14:26:42.2478553Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2478714Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2479002Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2479159Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2479446Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2479569Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2479849Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2479995Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2480272Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2480420Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2480696Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2480834Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2481112Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2481263Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2481757Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 2. CUDA driver allocated memory was 2300575744 and is now 17467179008.
2025-12-04T14:26:42.2481884Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2482078Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2482444Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2482557Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2482769Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2482934Z [rank2]:E1204 14:17:56.469000 383173 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2482971Z dist init r=2, world=4
2025-12-04T14:26:42.2483309Z [rank0]:[W1204 14:17:56.071657582 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2483641Z [rank1]:[W1204 14:17:56.158543370 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2483971Z [rank3]:[W1204 14:17:56.198589805 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2484300Z [rank2]:[W1204 14:17:56.342044735 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2484340Z FAILED [22.9260s] [  9%]
2025-12-04T14:26:42.2484342Z 
2025-12-04T14:26:42.2484399Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2484498Z _____ TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda _____
2025-12-04T14:26:42.2484544Z Traceback (most recent call last):
2025-12-04T14:26:42.2484711Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2484755Z     self._join_processes(fn)
2025-12-04T14:26:42.2484928Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2484981Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2485159Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2485204Z     raise RuntimeError(error)
2025-12-04T14:26:42.2485284Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2485328Z Traceback (most recent call last):
2025-12-04T14:26:42.2485488Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2485531Z     getattr(self, test_name)()
2025-12-04T14:26:42.2485707Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2485752Z     fn()
2025-12-04T14:26:42.2485903Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2485943Z     method(*args, **kwargs)
2025-12-04T14:26:42.2486092Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2486156Z     method(*args, **kwargs)
2025-12-04T14:26:42.2486307Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2486345Z     with policy():
2025-12-04T14:26:42.2486495Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2486537Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2486927Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 0. CUDA driver allocated memory was 2453667840 and is now 17620271104.
2025-12-04T14:26:42.2486931Z 
2025-12-04T14:26:42.2487007Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2487232Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2487236Z 
2025-12-04T14:26:42.2487324Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2487326Z 
2025-12-04T14:26:42.2487327Z 
2025-12-04T14:26:42.2487403Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2487490Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2487726Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d8feba8dedb219c1.xml -
2025-12-04T14:26:42.2487786Z =========================== short test summary info ============================
2025-12-04T14:26:42.2488030Z FAILED [22.9260s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2488076Z Traceback (most recent call last):
2025-12-04T14:26:42.2488242Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2488283Z     getattr(self, test_name)()
2025-12-04T14:26:42.2488443Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2488478Z     fn()
2025-12-04T14:26:42.2488629Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2488668Z     method(*args, **kwargs)
2025-12-04T14:26:42.2488819Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2488857Z     method(*args, **kwargs)
2025-12-04T14:26:42.2489007Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2489044Z     with policy():
2025-12-04T14:26:42.2489196Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2489237Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2489614Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 0. CUDA driver allocated memory was 2453667840 and is now 17620271104.
2025-12-04T14:26:42.2489630Z 
2025-12-04T14:26:42.2489703Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2489928Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2489930Z 
2025-12-04T14:26:42.2490031Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2490092Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2490153Z ======================= 1 failed, 9 deselected in 23.09s =======================
2025-12-04T14:26:42.2490189Z Got exit code 1
2025-12-04T14:26:42.2490230Z Retrying single test...
2025-12-04T14:26:42.2490419Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-e7f66684b4694643.xml
2025-12-04T14:26:42.2490478Z ============================= test session starts ==============================
2025-12-04T14:26:42.2490591Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2490632Z cachedir: .pytest_cache
2025-12-04T14:26:42.2490789Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2490836Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2490875Z configfile: pytest.ini
2025-12-04T14:26:42.2491037Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2491110Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2491330Z stepcurrent: skipping 9 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2491373Z Running 1 items in this shard
2025-12-04T14:26:42.2491375Z 
2025-12-04T14:26:42.2491676Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda I1204 14:18:13.210000 384512 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 384581
2025-12-04T14:26:42.2491832Z I1204 14:18:13.211000 384512 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 384582
2025-12-04T14:26:42.2491984Z I1204 14:18:13.211000 384512 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 384583
2025-12-04T14:26:42.2492133Z I1204 14:18:13.212000 384512 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 384584
2025-12-04T14:26:42.2492712Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2492750Z   _warn_cpu_init()
2025-12-04T14:26:42.2493047Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2493085Z   _init_core_state(
2025-12-04T14:26:42.2493599Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2493661Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2494244Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2494292Z   _warn_cpu_init()
2025-12-04T14:26:42.2494585Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2494624Z   _init_core_state(
2025-12-04T14:26:42.2495113Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2495175Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2495750Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2495786Z   _warn_cpu_init()
2025-12-04T14:26:42.2496079Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2496116Z   _init_core_state(
2025-12-04T14:26:42.2496606Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2496665Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2497281Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2497319Z   _warn_cpu_init()
2025-12-04T14:26:42.2497808Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2497866Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2498377Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2498448Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2498741Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2498793Z   _init_core_state(
2025-12-04T14:26:42.2499277Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2499333Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2499621Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2499662Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2500152Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2500210Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2500438Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2500481Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2500702Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2500744Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2500965Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2501005Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2501226Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2501266Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2501483Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2501524Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2501741Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2501782Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2501999Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2502039Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2502279Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2502320Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2502484Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2502647Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2502938Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2503106Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2503393Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2503516Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2503794Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2503942Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2504221Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2504370Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2504649Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2504788Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2505063Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2505214Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2505690Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 0. CUDA driver allocated memory was 2453667840 and is now 17620271104.
2025-12-04T14:26:42.2505807Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2506004Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2506358Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2506497Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2506710Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2506929Z [rank0]:E1204 14:18:21.902000 384581 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2506967Z dist init r=0, world=4
2025-12-04T14:26:42.2507107Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2507282Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2507568Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2507723Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2508008Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2508132Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2508409Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2508556Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2508833Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2508981Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2509258Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2509395Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2509673Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2509821Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2510293Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 3. CUDA driver allocated memory was 2250244096 and is now 17416847360.
2025-12-04T14:26:42.2510409Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2510604Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2510983Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2511108Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2511321Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2511496Z [rank3]:E1204 14:18:21.913000 384584 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2511534Z dist init r=3, world=4
2025-12-04T14:26:42.2511672Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2511834Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2512122Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2512276Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2512561Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2512686Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2512965Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2513112Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2513389Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2513536Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2513813Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2513951Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2514228Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2514378Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2514848Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 2. CUDA driver allocated memory was 2300575744 and is now 17467179008.
2025-12-04T14:26:42.2514963Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2515181Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2515545Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2515669Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2515879Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2516043Z [rank2]:E1204 14:18:21.915000 384583 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2516082Z dist init r=2, world=4
2025-12-04T14:26:42.2516221Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2516382Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2516669Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2516861Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2517146Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2517269Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2517546Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2517693Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2517968Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2518116Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2518392Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2518529Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2518808Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2518956Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2519460Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 1. CUDA driver allocated memory was 2317352960 and is now 17483956224.
2025-12-04T14:26:42.2519587Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2519782Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2520148Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2520261Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2520473Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2520636Z [rank1]:E1204 14:18:21.919000 384582 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2520675Z dist init r=1, world=4
2025-12-04T14:26:42.2521012Z [rank0]:[W1204 14:18:22.652317695 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2521342Z [rank3]:[W1204 14:18:22.676258828 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2521670Z [rank1]:[W1204 14:18:22.696432193 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2521996Z [rank2]:[W1204 14:18:22.725774866 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2522038Z FAILED [23.0261s] [100%]
2025-12-04T14:26:42.2522041Z 
2025-12-04T14:26:42.2522096Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2522196Z _____ TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda _____
2025-12-04T14:26:42.2522242Z Traceback (most recent call last):
2025-12-04T14:26:42.2522407Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2522450Z     self._join_processes(fn)
2025-12-04T14:26:42.2522622Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2522676Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2522854Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2522898Z     raise RuntimeError(error)
2025-12-04T14:26:42.2522978Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2523022Z Traceback (most recent call last):
2025-12-04T14:26:42.2523183Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2523225Z     getattr(self, test_name)()
2025-12-04T14:26:42.2523404Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2523440Z     fn()
2025-12-04T14:26:42.2523589Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2523642Z     method(*args, **kwargs)
2025-12-04T14:26:42.2523792Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2523832Z     method(*args, **kwargs)
2025-12-04T14:26:42.2523995Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2524032Z     with policy():
2025-12-04T14:26:42.2524183Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2524223Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2524574Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 3. CUDA driver allocated memory was 2250244096 and is now 17416847360.
2025-12-04T14:26:42.2524577Z 
2025-12-04T14:26:42.2524653Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2524879Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2524883Z 
2025-12-04T14:26:42.2524971Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2524973Z 
2025-12-04T14:26:42.2524975Z 
2025-12-04T14:26:42.2525050Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2525136Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2525369Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-e7f66684b4694643.xml -
2025-12-04T14:26:42.2525430Z =========================== short test summary info ============================
2025-12-04T14:26:42.2525673Z FAILED [23.0261s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2525719Z Traceback (most recent call last):
2025-12-04T14:26:42.2525884Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2525926Z     getattr(self, test_name)()
2025-12-04T14:26:42.2526086Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2526119Z     fn()
2025-12-04T14:26:42.2526272Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2526311Z     method(*args, **kwargs)
2025-12-04T14:26:42.2526463Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2526501Z     method(*args, **kwargs)
2025-12-04T14:26:42.2526651Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2526688Z     with policy():
2025-12-04T14:26:42.2526873Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2526914Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2527293Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 3. CUDA driver allocated memory was 2250244096 and is now 17416847360.
2025-12-04T14:26:42.2527296Z 
2025-12-04T14:26:42.2527371Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2527611Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2527613Z 
2025-12-04T14:26:42.2527700Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2527780Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2527842Z ====================== 1 failed, 19 deselected in 23.18s =======================
2025-12-04T14:26:42.2527879Z Got exit code 1
2025-12-04T14:26:42.2527920Z Retrying single test...
2025-12-04T14:26:42.2528111Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-bc09140a3bba9061.xml
2025-12-04T14:26:42.2528173Z ============================= test session starts ==============================
2025-12-04T14:26:42.2528284Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2528327Z cachedir: .pytest_cache
2025-12-04T14:26:42.2528484Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2528532Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2528573Z configfile: pytest.ini
2025-12-04T14:26:42.2528736Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2528812Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2529033Z stepcurrent: skipping 9 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2529080Z Running 1 items in this shard
2025-12-04T14:26:42.2529082Z 
2025-12-04T14:26:42.2529382Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda I1204 14:18:38.554000 385922 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 385991
2025-12-04T14:26:42.2529539Z I1204 14:18:38.555000 385922 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 385992
2025-12-04T14:26:42.2529691Z I1204 14:18:38.556000 385922 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 385993
2025-12-04T14:26:42.2529843Z I1204 14:18:38.556000 385922 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 385994
2025-12-04T14:26:42.2530422Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2530463Z   _warn_cpu_init()
2025-12-04T14:26:42.2530760Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2530798Z   _init_core_state(
2025-12-04T14:26:42.2531310Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2531372Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2531953Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2532002Z   _warn_cpu_init()
2025-12-04T14:26:42.2532297Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2532335Z   _init_core_state(
2025-12-04T14:26:42.2532829Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2532891Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2533459Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2533498Z   _warn_cpu_init()
2025-12-04T14:26:42.2533794Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2533831Z   _init_core_state(
2025-12-04T14:26:42.2534324Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2534383Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2534953Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2534991Z   _warn_cpu_init()
2025-12-04T14:26:42.2535481Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2535541Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2536056Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2536125Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2536416Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2025-12-04T14:26:42.2536467Z   _init_core_state(
2025-12-04T14:26:42.2537004Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2537061Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2537353Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2537395Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2537882Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2537939Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2538172Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2538214Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2538439Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2538481Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2538701Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2538744Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2538963Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2539004Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2539223Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2539267Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2539484Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2539525Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2539743Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2539784Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2540003Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2540071Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2540217Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2540395Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2540686Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2540855Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2541144Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2541268Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2541547Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2541695Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2541972Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2542122Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2542402Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2542541Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2542817Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2542967Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2543445Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 2. CUDA driver allocated memory was 2300575744 and is now 17467179008.
2025-12-04T14:26:42.2543563Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2543760Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2544114Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2544229Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2544464Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2544642Z [rank2]:E1204 14:18:47.279000 385993 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2544680Z dist init r=2, world=4
2025-12-04T14:26:42.2544819Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2544989Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2545280Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2545437Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2545721Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2545850Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2546128Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2546277Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2546555Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2546704Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2547034Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2547171Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2547449Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2547599Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2548071Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 0. CUDA driver allocated memory was 2453667840 and is now 17620271104.
2025-12-04T14:26:42.2548187Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2548382Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2548766Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2548878Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2549105Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2549271Z [rank0]:E1204 14:18:47.314000 385991 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2549332Z dist init r=0, world=4
2025-12-04T14:26:42.2549469Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2549631Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2549919Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2550072Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2550358Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2550483Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2550764Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2550911Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2551188Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2551335Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2551614Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2551751Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2552028Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2552178Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2552650Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 1. CUDA driver allocated memory was 2317352960 and is now 17483956224.
2025-12-04T14:26:42.2552765Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2552981Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2553333Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2553456Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2553680Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2553843Z [rank1]:E1204 14:18:47.327000 385992 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2553880Z dist init r=1, world=4
2025-12-04T14:26:42.2554020Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2554179Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2554470Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2554624Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2554909Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2555036Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2555312Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2555461Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2555737Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2555884Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2556162Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2556299Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2556578Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2556727Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2557268Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 3. CUDA driver allocated memory was 2250244096 and is now 17416847360.
2025-12-04T14:26:42.2557382Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2557592Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2557943Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2558071Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2558283Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2558446Z [rank3]:E1204 14:18:47.342000 385994 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2558485Z dist init r=3, world=4
2025-12-04T14:26:42.2558818Z [rank2]:[W1204 14:18:47.951370905 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2559149Z [rank0]:[W1204 14:18:47.056317075 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2559479Z [rank1]:[W1204 14:18:47.105902093 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2559804Z [rank3]:[W1204 14:18:47.207410621 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2559845Z FAILED [23.2264s] [100%]
2025-12-04T14:26:42.2559848Z 
2025-12-04T14:26:42.2559902Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2560002Z _____ TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda _____
2025-12-04T14:26:42.2560047Z Traceback (most recent call last):
2025-12-04T14:26:42.2560213Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2560257Z     self._join_processes(fn)
2025-12-04T14:26:42.2560431Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2560484Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2560663Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2560705Z     raise RuntimeError(error)
2025-12-04T14:26:42.2560786Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2560832Z Traceback (most recent call last):
2025-12-04T14:26:42.2560992Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2561036Z     getattr(self, test_name)()
2025-12-04T14:26:42.2561193Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2561251Z     fn()
2025-12-04T14:26:42.2561404Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2561458Z     method(*args, **kwargs)
2025-12-04T14:26:42.2561609Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2561650Z     method(*args, **kwargs)
2025-12-04T14:26:42.2561798Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2561849Z     with policy():
2025-12-04T14:26:42.2562001Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2562043Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2562393Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 0. CUDA driver allocated memory was 2453667840 and is now 17620271104.
2025-12-04T14:26:42.2562396Z 
2025-12-04T14:26:42.2562472Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2562699Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2562702Z 
2025-12-04T14:26:42.2562791Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2562794Z 
2025-12-04T14:26:42.2562856Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2562901Z Traceback (most recent call last):
2025-12-04T14:26:42.2563063Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2563105Z     getattr(self, test_name)()
2025-12-04T14:26:42.2563268Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2563302Z     fn()
2025-12-04T14:26:42.2563455Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2563495Z     method(*args, **kwargs)
2025-12-04T14:26:42.2563647Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2563686Z     method(*args, **kwargs)
2025-12-04T14:26:42.2563836Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2563872Z     with policy():
2025-12-04T14:26:42.2564023Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2564063Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2564412Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 2. CUDA driver allocated memory was 2300575744 and is now 17467179008.
2025-12-04T14:26:42.2564415Z 
2025-12-04T14:26:42.2564487Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2564714Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2564718Z 
2025-12-04T14:26:42.2564806Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2564808Z 
2025-12-04T14:26:42.2564810Z 
2025-12-04T14:26:42.2564885Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2564996Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2565229Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-bc09140a3bba9061.xml -
2025-12-04T14:26:42.2565310Z =========================== short test summary info ============================
2025-12-04T14:26:42.2565552Z FAILED [23.2264s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2565610Z Traceback (most recent call last):
2025-12-04T14:26:42.2565772Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2565814Z     getattr(self, test_name)()
2025-12-04T14:26:42.2565973Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2566009Z     fn()
2025-12-04T14:26:42.2566161Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2566203Z     method(*args, **kwargs)
2025-12-04T14:26:42.2566353Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2566393Z     method(*args, **kwargs)
2025-12-04T14:26:42.2566542Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2566580Z     with policy():
2025-12-04T14:26:42.2566730Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2566810Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2567159Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 0. CUDA driver allocated memory was 2453667840 and is now 17620271104.
2025-12-04T14:26:42.2567162Z 
2025-12-04T14:26:42.2567234Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2567459Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2567461Z 
2025-12-04T14:26:42.2567549Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2567551Z 
2025-12-04T14:26:42.2567612Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2567657Z Traceback (most recent call last):
2025-12-04T14:26:42.2567819Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2567860Z     getattr(self, test_name)()
2025-12-04T14:26:42.2568020Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2568057Z     fn()
2025-12-04T14:26:42.2568206Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2568245Z     method(*args, **kwargs)
2025-12-04T14:26:42.2568394Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2568434Z     method(*args, **kwargs)
2025-12-04T14:26:42.2568583Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2568619Z     with policy():
2025-12-04T14:26:42.2568769Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2568812Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2569183Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 80384 on device 2. CUDA driver allocated memory was 2300575744 and is now 17467179008.
2025-12-04T14:26:42.2569197Z 
2025-12-04T14:26:42.2569271Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2569494Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2569512Z 
2025-12-04T14:26:42.2569599Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2569661Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2569724Z ====================== 1 failed, 19 deselected in 23.39s =======================
2025-12-04T14:26:42.2569762Z Got exit code 1
2025-12-04T14:26:42.2569941Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda
2025-12-04T14:26:42.2570071Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2570260Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-2f4e8b75d9a22507.xml
2025-12-04T14:26:42.2570317Z ============================= test session starts ==============================
2025-12-04T14:26:42.2570430Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2570472Z cachedir: .pytest_cache
2025-12-04T14:26:42.2570629Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2570676Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2570719Z configfile: pytest.ini
2025-12-04T14:26:42.2570882Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2570957Z collecting ... collected 60 items / 10 deselected / 50 selected
2025-12-04T14:26:42.2571014Z stepcurrent: skipping 10 already run items.
2025-12-04T14:26:42.2571055Z Running 10 items in this shard
2025-12-04T14:26:42.2571057Z 
2025-12-04T14:26:42.2571397Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda I1204 14:19:04.223000 387332 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 387401
2025-12-04T14:26:42.2571553Z I1204 14:19:04.223000 387332 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 387402
2025-12-04T14:26:42.2571708Z I1204 14:19:04.224000 387332 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 387403
2025-12-04T14:26:42.2571860Z I1204 14:19:04.225000 387332 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 387404
2025-12-04T14:26:42.2572442Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2572481Z   _warn_cpu_init()
2025-12-04T14:26:42.2573000Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2573078Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2573650Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2573696Z   _warn_cpu_init()
2025-12-04T14:26:42.2574184Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2574244Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2574811Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2574849Z   _warn_cpu_init()
2025-12-04T14:26:42.2575339Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2575399Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2575965Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2576002Z   _warn_cpu_init()
2025-12-04T14:26:42.2576294Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2576377Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2576913Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2576971Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2577261Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2577369Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2577861Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2577933Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2578236Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2578315Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2578601Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2578677Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2579168Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2579228Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2579512Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2579594Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2579879Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2579953Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2580242Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2580322Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2580812Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2580871Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2581156Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2581232Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2582538Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2582685Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2582914Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2582956Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2584223Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2584349Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2584577Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2584622Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2585886Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2586010Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2586239Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2586279Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2587618Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2587767Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2587994Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2588036Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2588258Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2588299Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2588518Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2588560Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2588778Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2588817Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2589038Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2589079Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2589369Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2589409Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2589558Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2589721Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2590014Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2590170Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2590458Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2590586Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2590865Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2591036Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2591313Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2591476Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2591751Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2591905Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2592188Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2592337Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2592855Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 1. CUDA driver allocated memory was 2317352960 and is now 17494441984.
2025-12-04T14:26:42.2592972Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2593171Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2593565Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2593680Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2593894Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2594062Z [rank1]:E1204 14:19:36.163000 387402 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2594103Z dist init r=1, world=4
2025-12-04T14:26:42.2594244Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2594409Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2594695Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2594847Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2595133Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2595280Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2595559Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2595718Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2595995Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2596152Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2596432Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2596571Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2596893Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2597042Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2597554Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 3. CUDA driver allocated memory was 2250244096 and is now 17427333120.
2025-12-04T14:26:42.2597670Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2597867Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2598259Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2598373Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2598587Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2598753Z [rank3]:E1204 14:19:36.172000 387404 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2598793Z dist init r=3, world=4
2025-12-04T14:26:42.2598933Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2599093Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2599382Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2599534Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2599854Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2599994Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2600272Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2600438Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2600714Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2600860Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2601138Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2601276Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2601555Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2601703Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2602215Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 2. CUDA driver allocated memory was 2300575744 and is now 17477664768.
2025-12-04T14:26:42.2602329Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2602526Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2602918Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2603034Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2603248Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2603411Z [rank2]:E1204 14:19:36.210000 387403 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2603452Z dist init r=2, world=4
2025-12-04T14:26:42.2603590Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2603749Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2604057Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2604221Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2604506Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2604642Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2604919Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2605067Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2605345Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2605491Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2605770Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2605904Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2606183Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2606333Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2606884Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 0. CUDA driver allocated memory was 2453667840 and is now 17630756864.
2025-12-04T14:26:42.2607000Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2607196Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2607585Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2607699Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2607912Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2608076Z [rank0]:E1204 14:19:36.241000 387401 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2608114Z dist init r=0, world=4
2025-12-04T14:26:42.2608479Z [rank1]:[W1204 14:19:36.819253900 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2608828Z [rank3]:[W1204 14:19:36.853974896 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2609171Z [rank2]:[W1204 14:19:36.969901296 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2609502Z [rank0]:[W1204 14:19:36.014546897 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2609543Z FAILED [46.1455s] [ 10%]
2025-12-04T14:26:42.2609545Z 
2025-12-04T14:26:42.2609604Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2609731Z _ TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2609777Z Traceback (most recent call last):
2025-12-04T14:26:42.2609941Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2609987Z     self._join_processes(fn)
2025-12-04T14:26:42.2610160Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2610215Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2610394Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2610439Z     raise RuntimeError(error)
2025-12-04T14:26:42.2610520Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2610565Z Traceback (most recent call last):
2025-12-04T14:26:42.2610725Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2610769Z     getattr(self, test_name)()
2025-12-04T14:26:42.2610927Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2610963Z     fn()
2025-12-04T14:26:42.2611114Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2611155Z     method(*args, **kwargs)
2025-12-04T14:26:42.2611307Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2611347Z     method(*args, **kwargs)
2025-12-04T14:26:42.2611498Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2611537Z     with policy():
2025-12-04T14:26:42.2611691Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2611733Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2612122Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 1. CUDA driver allocated memory was 2317352960 and is now 17494441984.
2025-12-04T14:26:42.2612125Z 
2025-12-04T14:26:42.2612223Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2612489Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2612501Z 
2025-12-04T14:26:42.2612590Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2612592Z 
2025-12-04T14:26:42.2612594Z 
2025-12-04T14:26:42.2612671Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2612778Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2613014Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-2f4e8b75d9a22507.xml -
2025-12-04T14:26:42.2613074Z =========================== short test summary info ============================
2025-12-04T14:26:42.2613355Z FAILED [46.1455s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2613403Z Traceback (most recent call last):
2025-12-04T14:26:42.2613565Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2613608Z     getattr(self, test_name)()
2025-12-04T14:26:42.2613766Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2613803Z     fn()
2025-12-04T14:26:42.2613953Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2613993Z     method(*args, **kwargs)
2025-12-04T14:26:42.2614144Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2614184Z     method(*args, **kwargs)
2025-12-04T14:26:42.2614332Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2614371Z     with policy():
2025-12-04T14:26:42.2614521Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2614562Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2614948Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 1. CUDA driver allocated memory was 2317352960 and is now 17494441984.
2025-12-04T14:26:42.2614952Z 
2025-12-04T14:26:42.2615025Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2615291Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2615294Z 
2025-12-04T14:26:42.2615382Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2615445Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2615508Z ====================== 1 failed, 10 deselected in 46.31s =======================
2025-12-04T14:26:42.2615547Z Got exit code 1
2025-12-04T14:26:42.2615587Z Retrying single test...
2025-12-04T14:26:42.2615777Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-fd4fe51605acf2f2.xml
2025-12-04T14:26:42.2615834Z ============================= test session starts ==============================
2025-12-04T14:26:42.2615969Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2616010Z cachedir: .pytest_cache
2025-12-04T14:26:42.2616168Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2616228Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2616269Z configfile: pytest.ini
2025-12-04T14:26:42.2616431Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2616519Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2616841Z stepcurrent: skipping 10 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2616884Z Running 1 items in this shard
2025-12-04T14:26:42.2616886Z 
2025-12-04T14:26:42.2617225Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda I1204 14:19:52.851000 388598 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 388667
2025-12-04T14:26:42.2617382Z I1204 14:19:52.852000 388598 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 388668
2025-12-04T14:26:42.2617536Z I1204 14:19:52.852000 388598 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 388669
2025-12-04T14:26:42.2617687Z I1204 14:19:52.853000 388598 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 388670
2025-12-04T14:26:42.2618271Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2618310Z   _warn_cpu_init()
2025-12-04T14:26:42.2618811Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2618874Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2619447Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2619486Z   _warn_cpu_init()
2025-12-04T14:26:42.2619977Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2620038Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2620636Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2620686Z   _warn_cpu_init()
2025-12-04T14:26:42.2621176Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2621248Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2621817Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2621855Z   _warn_cpu_init()
2025-12-04T14:26:42.2622144Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2622232Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2622517Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2622599Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2623091Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2623150Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2623641Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2623700Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2623989Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2624069Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2624558Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2624618Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2624925Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2625004Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2625301Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2625377Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2625671Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2625745Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2626237Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2626295Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2626581Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2626660Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2626986Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2627060Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2628346Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2628474Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2628705Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2628749Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2630042Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2630185Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2631466Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2631590Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2631817Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2631860Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2632086Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2632127Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2633392Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2633514Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2633738Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2633781Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2634001Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2634042Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2634282Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2634325Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2634558Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2634599Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2634818Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2634871Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2635163Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2635203Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2635348Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2635512Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2635802Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2635959Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2636244Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2636369Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2636648Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2636837Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2637117Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2637268Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2637547Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2637687Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2637966Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2638116Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2638655Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 2. CUDA driver allocated memory was 2300575744 and is now 17477664768.
2025-12-04T14:26:42.2638786Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2638983Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2639387Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2639504Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2639717Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2639882Z [rank2]:E1204 14:20:24.948000 388669 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2639921Z dist init r=2, world=4
2025-12-04T14:26:42.2640060Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2640223Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2640510Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2640668Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2640952Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2641078Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2641354Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2641503Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2641779Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2641928Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2642204Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2642340Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2642621Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2642789Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2643300Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 0. CUDA driver allocated memory was 2453667840 and is now 17630756864.
2025-12-04T14:26:42.2643436Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2643632Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2644023Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2644137Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2644349Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2644515Z [rank0]:E1204 14:20:24.958000 388667 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2644554Z dist init r=0, world=4
2025-12-04T14:26:42.2644692Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2644853Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2645141Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2645296Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2645584Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2645708Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2645987Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2646135Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2646413Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2646564Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2646874Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2647043Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2647321Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2647483Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2647994Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 3. CUDA driver allocated memory was 2250244096 and is now 17427333120.
2025-12-04T14:26:42.2648128Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2648325Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2648713Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2648827Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2649038Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2649204Z [rank3]:E1204 14:20:24.969000 388670 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2649242Z dist init r=3, world=4
2025-12-04T14:26:42.2649379Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2649538Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2649826Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2649980Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2650265Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2650388Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2650665Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2650813Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2651088Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2651236Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2651531Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2651681Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2651958Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2652117Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2652627Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 1. CUDA driver allocated memory was 2317352960 and is now 17494441984.
2025-12-04T14:26:42.2652742Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2652937Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2653331Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2653445Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2653657Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2653822Z [rank1]:E1204 14:20:25.013000 388668 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2653861Z dist init r=1, world=4
2025-12-04T14:26:42.2654194Z [rank2]:[W1204 14:20:25.604794494 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2654525Z [rank0]:[W1204 14:20:25.629878199 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2654851Z [rank3]:[W1204 14:20:25.647113454 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2655179Z [rank1]:[W1204 14:20:25.782395696 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2655222Z FAILED [46.4449s] [100%]
2025-12-04T14:26:42.2655224Z 
2025-12-04T14:26:42.2655280Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2655410Z _ TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2655455Z Traceback (most recent call last):
2025-12-04T14:26:42.2655644Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2655699Z     self._join_processes(fn)
2025-12-04T14:26:42.2655875Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2655929Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2656108Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2656165Z     raise RuntimeError(error)
2025-12-04T14:26:42.2656245Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2656291Z Traceback (most recent call last):
2025-12-04T14:26:42.2656451Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2656495Z     getattr(self, test_name)()
2025-12-04T14:26:42.2656653Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2656689Z     fn()
2025-12-04T14:26:42.2656874Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2656915Z     method(*args, **kwargs)
2025-12-04T14:26:42.2657065Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2657107Z     method(*args, **kwargs)
2025-12-04T14:26:42.2657257Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2657294Z     with policy():
2025-12-04T14:26:42.2657444Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2657488Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2657873Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 0. CUDA driver allocated memory was 2453667840 and is now 17630756864.
2025-12-04T14:26:42.2657877Z 
2025-12-04T14:26:42.2657953Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2658221Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2658224Z 
2025-12-04T14:26:42.2658311Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2658313Z 
2025-12-04T14:26:42.2658373Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2658420Z Traceback (most recent call last):
2025-12-04T14:26:42.2658583Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2658626Z     getattr(self, test_name)()
2025-12-04T14:26:42.2658786Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2658819Z     fn()
2025-12-04T14:26:42.2658970Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2659010Z     method(*args, **kwargs)
2025-12-04T14:26:42.2659159Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2659198Z     method(*args, **kwargs)
2025-12-04T14:26:42.2659347Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2659421Z     with policy():
2025-12-04T14:26:42.2659574Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2659629Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2660011Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 2. CUDA driver allocated memory was 2300575744 and is now 17477664768.
2025-12-04T14:26:42.2660031Z 
2025-12-04T14:26:42.2660105Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2660368Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2660370Z 
2025-12-04T14:26:42.2660459Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2660461Z 
2025-12-04T14:26:42.2660519Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2660565Z Traceback (most recent call last):
2025-12-04T14:26:42.2660729Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2660771Z     getattr(self, test_name)()
2025-12-04T14:26:42.2660930Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2660966Z     fn()
2025-12-04T14:26:42.2661115Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2661155Z     method(*args, **kwargs)
2025-12-04T14:26:42.2661304Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2661345Z     method(*args, **kwargs)
2025-12-04T14:26:42.2661495Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2661534Z     with policy():
2025-12-04T14:26:42.2661683Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2661724Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2662106Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 3. CUDA driver allocated memory was 2250244096 and is now 17427333120.
2025-12-04T14:26:42.2662110Z 
2025-12-04T14:26:42.2662182Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2662450Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2662453Z 
2025-12-04T14:26:42.2662539Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2662541Z 
2025-12-04T14:26:42.2662543Z 
2025-12-04T14:26:42.2662619Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2662706Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2662942Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-fd4fe51605acf2f2.xml -
2025-12-04T14:26:42.2663004Z =========================== short test summary info ============================
2025-12-04T14:26:42.2663311Z FAILED [46.4449s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2663359Z Traceback (most recent call last):
2025-12-04T14:26:42.2663537Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2663580Z     getattr(self, test_name)()
2025-12-04T14:26:42.2663738Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2663784Z     fn()
2025-12-04T14:26:42.2663934Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2663974Z     method(*args, **kwargs)
2025-12-04T14:26:42.2664123Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2664162Z     method(*args, **kwargs)
2025-12-04T14:26:42.2664313Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2664350Z     with policy():
2025-12-04T14:26:42.2664500Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2664542Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2664928Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 0. CUDA driver allocated memory was 2453667840 and is now 17630756864.
2025-12-04T14:26:42.2664933Z 
2025-12-04T14:26:42.2665005Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2665269Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2665272Z 
2025-12-04T14:26:42.2665358Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2665361Z 
2025-12-04T14:26:42.2665419Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2665464Z Traceback (most recent call last):
2025-12-04T14:26:42.2665628Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2665670Z     getattr(self, test_name)()
2025-12-04T14:26:42.2665829Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2665862Z     fn()
2025-12-04T14:26:42.2666014Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2666052Z     method(*args, **kwargs)
2025-12-04T14:26:42.2666204Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2666242Z     method(*args, **kwargs)
2025-12-04T14:26:42.2666394Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2666430Z     with policy():
2025-12-04T14:26:42.2666583Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2668531Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2668919Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 2. CUDA driver allocated memory was 2300575744 and is now 17477664768.
2025-12-04T14:26:42.2668922Z 
2025-12-04T14:26:42.2669035Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2669297Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2669318Z 
2025-12-04T14:26:42.2669405Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2669407Z 
2025-12-04T14:26:42.2669467Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2671464Z Traceback (most recent call last):
2025-12-04T14:26:42.2671635Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2671677Z     getattr(self, test_name)()
2025-12-04T14:26:42.2671840Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2671874Z     fn()
2025-12-04T14:26:42.2672029Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2672071Z     method(*args, **kwargs)
2025-12-04T14:26:42.2672222Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2672260Z     method(*args, **kwargs)
2025-12-04T14:26:42.2672410Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2672466Z     with policy():
2025-12-04T14:26:42.2672617Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2672658Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2673047Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 3. CUDA driver allocated memory was 2250244096 and is now 17427333120.
2025-12-04T14:26:42.2673053Z 
2025-12-04T14:26:42.2673126Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2673388Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2673391Z 
2025-12-04T14:26:42.2673478Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2673543Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2673606Z ====================== 1 failed, 19 deselected in 46.61s =======================
2025-12-04T14:26:42.2673643Z Got exit code 1
2025-12-04T14:26:42.2673682Z Retrying single test...
2025-12-04T14:26:42.2673876Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-199eda7c5ce46bfe.xml
2025-12-04T14:26:42.2673933Z ============================= test session starts ==============================
2025-12-04T14:26:42.2674049Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2674089Z cachedir: .pytest_cache
2025-12-04T14:26:42.2674249Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2674297Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2674338Z configfile: pytest.ini
2025-12-04T14:26:42.2674503Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2674578Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2674853Z stepcurrent: skipping 10 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2674898Z Running 1 items in this shard
2025-12-04T14:26:42.2674911Z 
2025-12-04T14:26:42.2675251Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda I1204 14:20:41.638000 389864 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 389933
2025-12-04T14:26:42.2675493Z I1204 14:20:41.639000 389864 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 389934
2025-12-04T14:26:42.2675647Z I1204 14:20:41.640000 389864 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 389935
2025-12-04T14:26:42.2675798Z I1204 14:20:41.640000 389864 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 389936
2025-12-04T14:26:42.2676384Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2676423Z   _warn_cpu_init()
2025-12-04T14:26:42.2676961Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2677024Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2677597Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2677636Z   _warn_cpu_init()
2025-12-04T14:26:42.2678125Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2678187Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2678756Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2678794Z   _warn_cpu_init()
2025-12-04T14:26:42.2679283Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2679360Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2679931Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2680027Z   _warn_cpu_init()
2025-12-04T14:26:42.2680318Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2680402Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2680894Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2680953Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2681249Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2681328Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2681613Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2681693Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2682185Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2682244Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2682533Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2682609Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2682895Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2682976Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2683467Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2683526Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2683823Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2683899Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2684193Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:787: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2684274Z   shared = FSDP(shared, group, **fsdp_kwargs)  # type: ignore[assignment]
2025-12-04T14:26:42.2684791Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.2684850Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.2685141Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2685215Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.2686504Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2686633Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2687943Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2688068Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2688300Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2688344Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2688584Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2688627Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2689919Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2690058Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2690284Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2690327Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2691588Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2691710Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2691937Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2691977Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2692199Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2692240Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2692460Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2692501Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2692721Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2692762Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2692980Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2693020Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2693323Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2693374Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2693520Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2693696Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2693998Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2694155Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2694442Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2694569Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2694851Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2695003Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2695283Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2695433Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2695710Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2695849Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2696125Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2696275Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2696831Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 3. CUDA driver allocated memory was 2250244096 and is now 17427333120.
2025-12-04T14:26:42.2696950Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2697149Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2697556Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2697691Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2697904Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2698082Z [rank3]:E1204 14:21:13.701000 389936 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2698135Z dist init r=3, world=4
2025-12-04T14:26:42.2698276Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2698435Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2698723Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2698881Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2699165Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2699291Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2699568Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2699716Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2699993Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2700142Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2700421Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2700556Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2700834Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2700982Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2701493Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 2. CUDA driver allocated memory was 2300575744 and is now 17477664768.
2025-12-04T14:26:42.2701609Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2701817Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2702221Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2702356Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2702570Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2702734Z [rank2]:E1204 14:21:13.755000 389935 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2702775Z dist init r=2, world=4
2025-12-04T14:26:42.2702912Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2703210Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2703500Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2703654Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2703941Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2704064Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2704342Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2704489Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2704767Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2704914Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2705191Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2705329Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2705612Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2705762Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2706285Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 0. CUDA driver allocated memory was 2453667840 and is now 17630756864.
2025-12-04T14:26:42.2706412Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2706608Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2707064Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2707180Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2707393Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2707560Z [rank0]:E1204 14:21:13.766000 389933 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2707598Z dist init r=0, world=4
2025-12-04T14:26:42.2707737Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2707898Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2708185Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2708340Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2708626Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2708751Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2709028Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2709176Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2709452Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2709600Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2709876Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2710013Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2710293Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2710457Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2710974Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 1. CUDA driver allocated memory was 2317352960 and is now 17494441984.
2025-12-04T14:26:42.2711130Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2711327Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2711720Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2711834Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2712045Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2712210Z [rank1]:E1204 14:21:13.767000 389934 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2712250Z dist init r=1, world=4
2025-12-04T14:26:42.2712587Z [rank3]:[W1204 14:21:13.358847842 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2712918Z [rank0]:[W1204 14:21:14.522013615 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2713248Z [rank2]:[W1204 14:21:14.522017145 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2713574Z [rank1]:[W1204 14:21:14.576527734 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2713616Z FAILED [46.3464s] [100%]
2025-12-04T14:26:42.2713618Z 
2025-12-04T14:26:42.2713675Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2713805Z _ TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2713852Z Traceback (most recent call last):
2025-12-04T14:26:42.2714017Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2714060Z     self._join_processes(fn)
2025-12-04T14:26:42.2714236Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2714289Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2714470Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2714514Z     raise RuntimeError(error)
2025-12-04T14:26:42.2714612Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2714658Z Traceback (most recent call last):
2025-12-04T14:26:42.2714831Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2714875Z     getattr(self, test_name)()
2025-12-04T14:26:42.2715032Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2715081Z     fn()
2025-12-04T14:26:42.2715243Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2715286Z     method(*args, **kwargs)
2025-12-04T14:26:42.2715435Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2715476Z     method(*args, **kwargs)
2025-12-04T14:26:42.2715627Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2715665Z     with policy():
2025-12-04T14:26:42.2715816Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2715858Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2716248Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 3. CUDA driver allocated memory was 2250244096 and is now 17427333120.
2025-12-04T14:26:42.2716251Z 
2025-12-04T14:26:42.2716327Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2716591Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2716595Z 
2025-12-04T14:26:42.2716681Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2716685Z 
2025-12-04T14:26:42.2716686Z 
2025-12-04T14:26:42.2716814Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2716902Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2717139Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-199eda7c5ce46bfe.xml -
2025-12-04T14:26:42.2717200Z =========================== short test summary info ============================
2025-12-04T14:26:42.2717480Z FAILED [46.3464s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2717526Z Traceback (most recent call last):
2025-12-04T14:26:42.2717689Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2717732Z     getattr(self, test_name)()
2025-12-04T14:26:42.2717892Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2717926Z     fn()
2025-12-04T14:26:42.2718078Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2718119Z     method(*args, **kwargs)
2025-12-04T14:26:42.2718270Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2718310Z     method(*args, **kwargs)
2025-12-04T14:26:42.2718460Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2718512Z     with policy():
2025-12-04T14:26:42.2718664Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2718720Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2719117Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 215552 on device 3. CUDA driver allocated memory was 2250244096 and is now 17427333120.
2025-12-04T14:26:42.2719135Z 
2025-12-04T14:26:42.2719209Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2719470Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2719473Z 
2025-12-04T14:26:42.2719561Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2719624Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2719688Z ====================== 1 failed, 19 deselected in 46.50s =======================
2025-12-04T14:26:42.2719725Z Got exit code 1
2025-12-04T14:26:42.2719937Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda
2025-12-04T14:26:42.2720069Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2720258Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-cbb13607d1190c6a.xml
2025-12-04T14:26:42.2720315Z ============================= test session starts ==============================
2025-12-04T14:26:42.2720428Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2720470Z cachedir: .pytest_cache
2025-12-04T14:26:42.2720629Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2720677Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2720718Z configfile: pytest.ini
2025-12-04T14:26:42.2720882Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2720958Z collecting ... collected 60 items / 11 deselected / 49 selected
2025-12-04T14:26:42.2721012Z stepcurrent: skipping 11 already run items.
2025-12-04T14:26:42.2721056Z Running 9 items in this shard
2025-12-04T14:26:42.2721058Z 
2025-12-04T14:26:42.2721383Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda I1204 14:21:30.384000 391130 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 391199
2025-12-04T14:26:42.2721537Z I1204 14:21:30.385000 391130 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 391200
2025-12-04T14:26:42.2721691Z I1204 14:21:30.385000 391130 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 391201
2025-12-04T14:26:42.2721842Z I1204 14:21:30.386000 391130 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 391202
2025-12-04T14:26:42.2722131Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2722180Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2722769Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2722818Z   _warn_cpu_init()
2025-12-04T14:26:42.2723120Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2723223Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2723506Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2723552Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2723830Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2723875Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2724446Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2724483Z   _warn_cpu_init()
2025-12-04T14:26:42.2725048Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2725087Z   _warn_cpu_init()
2025-12-04T14:26:42.2725374Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2725463Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2725746Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2725832Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2726108Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2726154Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2726719Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2726796Z   _warn_cpu_init()
2025-12-04T14:26:42.2727099Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2727197Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2727428Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2727486Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2727732Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2727774Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2727999Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2728039Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2728260Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2728300Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2728520Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2728563Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2728781Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2728822Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2729040Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2729080Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2729299Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2729339Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2729630Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2729672Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2730949Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2731077Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2732361Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2732507Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2733772Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2733895Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2735159Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2735281Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2735426Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2735589Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2735881Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2736036Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2736333Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2736468Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2736805Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2736970Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2737249Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2737396Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2737677Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2737815Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2738094Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2738244Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2738736Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 193024 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2738852Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2739050Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2739420Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2739535Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2739749Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2739914Z [rank1]:E1204 14:21:37.867000 391200 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2739954Z dist init r=1, world=4
2025-12-04T14:26:42.2740093Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2740252Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2740557Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2740725Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2741009Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2741163Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2741440Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2741590Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2741864Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2742013Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2742293Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2742428Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2742707Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2742857Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2743348Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 188928 on device 3. CUDA driver allocated memory was 2250244096 and is now 3808428032.
2025-12-04T14:26:42.2743463Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2743661Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2744030Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2744145Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2744360Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2744525Z [rank3]:E1204 14:21:37.873000 391202 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2744563Z dist init r=3, world=4
2025-12-04T14:26:42.2744713Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2744873Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2745170Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2745374Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2745660Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2745786Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2746065Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2746213Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2746490Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2746638Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2746954Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2747092Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2747371Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2747521Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2748008Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 184832 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2748123Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2748319Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2748688Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2748803Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2749028Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2749192Z [rank2]:E1204 14:21:37.874000 391201 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2749244Z dist init r=2, world=4
2025-12-04T14:26:42.2749382Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2749541Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2749858Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2750011Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2750297Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2750422Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2750698Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2750847Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2751128Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2751278Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2751553Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2751689Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2751968Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2752116Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2752604Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 172544 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2752718Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2752915Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2753293Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2753409Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2753635Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2753799Z [rank0]:E1204 14:21:37.961000 391199 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2753851Z dist init r=0, world=4
2025-12-04T14:26:42.2754201Z [rank0]:[W1204 14:21:38.751072961 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2754241Z FAILED [9.3146s] [ 11%]
2025-12-04T14:26:42.2754243Z 
2025-12-04T14:26:42.2754300Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2754409Z _ TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2754456Z Traceback (most recent call last):
2025-12-04T14:26:42.2754620Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2754663Z     self._join_processes(fn)
2025-12-04T14:26:42.2754838Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2754893Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2755071Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2755115Z     raise RuntimeError(error)
2025-12-04T14:26:42.2755195Z RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2755241Z Traceback (most recent call last):
2025-12-04T14:26:42.2755403Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2755447Z     getattr(self, test_name)()
2025-12-04T14:26:42.2755604Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2755639Z     fn()
2025-12-04T14:26:42.2755790Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2755832Z     method(*args, **kwargs)
2025-12-04T14:26:42.2755983Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2756024Z     method(*args, **kwargs)
2025-12-04T14:26:42.2756174Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2756211Z     with policy():
2025-12-04T14:26:42.2756363Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2756405Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2756810Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 184832 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2756813Z 
2025-12-04T14:26:42.2756889Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2757131Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2757135Z 
2025-12-04T14:26:42.2757237Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2757239Z 
2025-12-04T14:26:42.2757241Z 
2025-12-04T14:26:42.2757318Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2757421Z Process 2 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2757655Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-cbb13607d1190c6a.xml -
2025-12-04T14:26:42.2757745Z =========================== short test summary info ============================
2025-12-04T14:26:42.2758005Z FAILED [9.3146s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda - RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2758051Z Traceback (most recent call last):
2025-12-04T14:26:42.2758216Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2758258Z     getattr(self, test_name)()
2025-12-04T14:26:42.2758418Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2758453Z     fn()
2025-12-04T14:26:42.2758604Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2758642Z     method(*args, **kwargs)
2025-12-04T14:26:42.2758795Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2758835Z     method(*args, **kwargs)
2025-12-04T14:26:42.2758984Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2759022Z     with policy():
2025-12-04T14:26:42.2759174Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2759215Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2759580Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 184832 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2759582Z 
2025-12-04T14:26:42.2759656Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2759899Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2759901Z 
2025-12-04T14:26:42.2759989Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2760051Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2760113Z ======================= 1 failed, 11 deselected in 9.48s =======================
2025-12-04T14:26:42.2760150Z Got exit code 1
2025-12-04T14:26:42.2760191Z Retrying single test...
2025-12-04T14:26:42.2760378Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-75401eb34135640f.xml
2025-12-04T14:26:42.2760436Z ============================= test session starts ==============================
2025-12-04T14:26:42.2760551Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2760591Z cachedir: .pytest_cache
2025-12-04T14:26:42.2760748Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2760793Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2760834Z configfile: pytest.ini
2025-12-04T14:26:42.2761007Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2761084Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2761339Z stepcurrent: skipping 11 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2761383Z Running 1 items in this shard
2025-12-04T14:26:42.2761385Z 
2025-12-04T14:26:42.2761730Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda I1204 14:21:42.162000 391532 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 391601
2025-12-04T14:26:42.2761887Z I1204 14:21:42.163000 391532 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 391602
2025-12-04T14:26:42.2762041Z I1204 14:21:42.163000 391532 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 391603
2025-12-04T14:26:42.2762192Z I1204 14:21:42.164000 391532 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 391604
2025-12-04T14:26:42.2762480Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2762527Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2763107Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2763144Z   _warn_cpu_init()
2025-12-04T14:26:42.2763426Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2763472Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2764043Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2764081Z   _warn_cpu_init()
2025-12-04T14:26:42.2764371Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2764460Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2764747Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2764835Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2765113Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2765158Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2765737Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2765785Z   _warn_cpu_init()
2025-12-04T14:26:42.2766075Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2766131Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2766703Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2766783Z   _warn_cpu_init()
2025-12-04T14:26:42.2767074Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2767161Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2767448Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2767533Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2767764Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2767808Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2768033Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2768076Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2768299Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2768341Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2768563Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2768604Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2768823Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2768865Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2769084Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2769126Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2769347Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2769387Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2769620Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2769661Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2769951Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2770006Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2771292Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2771432Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2772702Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2772826Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2774093Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2774217Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2775504Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2775648Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2775793Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2775956Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2776249Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2776406Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2776697Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2776863Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2777142Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2777292Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2777570Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2777718Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2777995Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2778133Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2778412Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2778561Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2779078Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 188928 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2779194Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2779405Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2779792Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2779919Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2780133Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2780297Z [rank1]:E1204 14:21:49.648000 391602 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2780439Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2780598Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2780888Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2781041Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2781330Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2781458Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2781736Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2781885Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2782161Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2782310Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2782585Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2782722Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2783002Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2783149Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2783650Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 193024 on device 3. CUDA driver allocated memory was 2250244096 and is now 3808428032.
2025-12-04T14:26:42.2783775Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2783993Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2784362Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2784477Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2784690Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2784853Z [rank3]:E1204 14:21:49.648000 391604 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2784894Z dist init r=3, world=4
2025-12-04T14:26:42.2784932Z dist init r=1, world=4
2025-12-04T14:26:42.2785071Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2785230Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2785520Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2785673Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2785959Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2786084Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2786361Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2786510Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2786824Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2786972Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2787250Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2787386Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2787677Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2787838Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2788341Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 182784 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2788470Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2788668Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2789037Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2789154Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2789368Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2789531Z [rank0]:E1204 14:21:49.698000 391601 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2789569Z dist init r=0, world=4
2025-12-04T14:26:42.2789707Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2789867Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2790158Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2790314Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2790601Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2790725Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2791004Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2791153Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2791433Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2791580Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2791876Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2792013Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2792303Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2792480Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2792966Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 188928 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2793081Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2793276Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2793643Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2793757Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2793970Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2794135Z [rank2]:E1204 14:21:49.712000 391603 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2794173Z dist init r=2, world=4
2025-12-04T14:26:42.2794509Z [rank0]:[W1204 14:21:49.446208474 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2794550Z FAILED [9.2155s] [100%]
2025-12-04T14:26:42.2794552Z 
2025-12-04T14:26:42.2794609Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2794717Z _ TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2794763Z Traceback (most recent call last):
2025-12-04T14:26:42.2794927Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2794971Z     self._join_processes(fn)
2025-12-04T14:26:42.2795144Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2795199Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2795376Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2795423Z     raise RuntimeError(error)
2025-12-04T14:26:42.2795502Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2795548Z Traceback (most recent call last):
2025-12-04T14:26:42.2795707Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2795750Z     getattr(self, test_name)()
2025-12-04T14:26:42.2795922Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2795956Z     fn()
2025-12-04T14:26:42.2796121Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2796161Z     method(*args, **kwargs)
2025-12-04T14:26:42.2796312Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2796363Z     method(*args, **kwargs)
2025-12-04T14:26:42.2796524Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2796561Z     with policy():
2025-12-04T14:26:42.2796714Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2796804Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2797169Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 193024 on device 3. CUDA driver allocated memory was 2250244096 and is now 3808428032.
2025-12-04T14:26:42.2797172Z 
2025-12-04T14:26:42.2797247Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2797491Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2797494Z 
2025-12-04T14:26:42.2797582Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2797585Z 
2025-12-04T14:26:42.2797586Z 
2025-12-04T14:26:42.2797661Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2797750Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2797983Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-75401eb34135640f.xml -
2025-12-04T14:26:42.2798045Z =========================== short test summary info ============================
2025-12-04T14:26:42.2798302Z FAILED [9.2155s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2798350Z Traceback (most recent call last):
2025-12-04T14:26:42.2798512Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2798555Z     getattr(self, test_name)()
2025-12-04T14:26:42.2798715Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2798754Z     fn()
2025-12-04T14:26:42.2798906Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2798950Z     method(*args, **kwargs)
2025-12-04T14:26:42.2799101Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2799142Z     method(*args, **kwargs)
2025-12-04T14:26:42.2799294Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2799333Z     with policy():
2025-12-04T14:26:42.2799483Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2799524Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2799907Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 193024 on device 3. CUDA driver allocated memory was 2250244096 and is now 3808428032.
2025-12-04T14:26:42.2799922Z 
2025-12-04T14:26:42.2799998Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2800241Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2800258Z 
2025-12-04T14:26:42.2800362Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2800428Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2800489Z ======================= 1 failed, 19 deselected in 9.38s =======================
2025-12-04T14:26:42.2800526Z Got exit code 1
2025-12-04T14:26:42.2800565Z Retrying single test...
2025-12-04T14:26:42.2800755Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4c729ae48df975c6.xml
2025-12-04T14:26:42.2800812Z ============================= test session starts ==============================
2025-12-04T14:26:42.2800927Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2800966Z cachedir: .pytest_cache
2025-12-04T14:26:42.2801125Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2801173Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2801214Z configfile: pytest.ini
2025-12-04T14:26:42.2801376Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2801450Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2801687Z stepcurrent: skipping 11 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2801731Z Running 1 items in this shard
2025-12-04T14:26:42.2801734Z 
2025-12-04T14:26:42.2802052Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda I1204 14:21:53.940000 391934 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 392003
2025-12-04T14:26:42.2802207Z I1204 14:21:53.940000 391934 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 392004
2025-12-04T14:26:42.2802362Z I1204 14:21:53.941000 391934 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 392005
2025-12-04T14:26:42.2802512Z I1204 14:21:53.942000 391934 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 392006
2025-12-04T14:26:42.2802803Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2802850Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2803130Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2803176Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2803768Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2803806Z   _warn_cpu_init()
2025-12-04T14:26:42.2804369Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2804446Z   _warn_cpu_init()
2025-12-04T14:26:42.2804728Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2804771Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2805343Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2805379Z   _warn_cpu_init()
2025-12-04T14:26:42.2805661Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2805705Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2806273Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2806311Z   _warn_cpu_init()
2025-12-04T14:26:42.2806598Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2806689Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2807024Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2807111Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2807396Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2807480Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2807769Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2807856Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2808086Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2808129Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2808371Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2808429Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2808651Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2808690Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2808950Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2808990Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2809209Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2809249Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2809469Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2809509Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2809728Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2809768Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2809988Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2810028Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2810321Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2810362Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2811634Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2811762Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2813042Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2813178Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2814449Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2814582Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2815852Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.2815976Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.2816120Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2816283Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2816575Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2816733Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2817058Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2817186Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2817484Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2817637Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2817931Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2818092Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2818385Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2818522Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2818805Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2818955Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2819448Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 180736 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2819565Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2819761Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2820137Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2820253Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2820467Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2820634Z [rank2]:E1204 14:22:01.460000 392005 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2820671Z dist init r=2, world=4
2025-12-04T14:26:42.2820812Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2820971Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2821260Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2821417Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2821705Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2821839Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2822118Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2822276Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2822575Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2822724Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2823002Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2823142Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2823419Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2823571Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2824059Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 176640 on device 0. CUDA driver allocated memory was 2453667840 and is now 4011851776.
2025-12-04T14:26:42.2824174Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2824369Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2824737Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2824852Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2825066Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2825232Z [rank0]:E1204 14:22:01.469000 392003 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2825273Z dist init r=0, world=4
2025-12-04T14:26:42.2825411Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2825571Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2825861Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2826029Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2826314Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2826455Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2826784Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2826949Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2827228Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2827374Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2827652Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2827789Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2828069Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2828219Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2828709Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 193024 on device 1. CUDA driver allocated memory was 2317352960 and is now 3875536896.
2025-12-04T14:26:42.2828825Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2829020Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2829389Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2829500Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2829714Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2829879Z [rank1]:E1204 14:22:01.475000 392004 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2829917Z dist init r=1, world=4
2025-12-04T14:26:42.2830057Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2830215Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2830522Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2830688Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2830987Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2831122Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2831401Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2831551Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2831830Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2831976Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2832252Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2832388Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2832666Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2832815Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2833304Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 180736 on device 3. CUDA driver allocated memory was 2250244096 and is now 3808428032.
2025-12-04T14:26:42.2833420Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2833622Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2833987Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2834104Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2834315Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2834480Z [rank3]:E1204 14:22:01.517000 392006 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2834517Z dist init r=3, world=4
2025-12-04T14:26:42.2834866Z [rank0]:[W1204 14:22:01.188023818 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2834917Z FAILED [9.4148s] [100%]
2025-12-04T14:26:42.2834920Z 
2025-12-04T14:26:42.2834975Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2835084Z _ TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda _
2025-12-04T14:26:42.2835153Z Traceback (most recent call last):
2025-12-04T14:26:42.2835319Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2835363Z     self._join_processes(fn)
2025-12-04T14:26:42.2835538Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2835591Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2835768Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2835811Z     raise RuntimeError(error)
2025-12-04T14:26:42.2835893Z RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2835938Z Traceback (most recent call last):
2025-12-04T14:26:42.2836101Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2836144Z     getattr(self, test_name)()
2025-12-04T14:26:42.2836303Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2836336Z     fn()
2025-12-04T14:26:42.2836489Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2836529Z     method(*args, **kwargs)
2025-12-04T14:26:42.2836682Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2836722Z     method(*args, **kwargs)
2025-12-04T14:26:42.2836915Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2836954Z     with policy():
2025-12-04T14:26:42.2837106Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2837148Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2837510Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 180736 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2837514Z 
2025-12-04T14:26:42.2837588Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2837829Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2837832Z 
2025-12-04T14:26:42.2837921Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2837923Z 
2025-12-04T14:26:42.2837926Z 
2025-12-04T14:26:42.2838001Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2838091Z Process 2 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2838326Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4c729ae48df975c6.xml -
2025-12-04T14:26:42.2838400Z =========================== short test summary info ============================
2025-12-04T14:26:42.2838659Z FAILED [9.4148s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda - RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.2838718Z Traceback (most recent call last):
2025-12-04T14:26:42.2838883Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2838924Z     getattr(self, test_name)()
2025-12-04T14:26:42.2839114Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2839149Z     fn()
2025-12-04T14:26:42.2839303Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2839341Z     method(*args, **kwargs)
2025-12-04T14:26:42.2839495Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2839534Z     method(*args, **kwargs)
2025-12-04T14:26:42.2839685Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2839721Z     with policy():
2025-12-04T14:26:42.2839873Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2839915Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2840278Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 180736 on device 2. CUDA driver allocated memory was 2300575744 and is now 3858759680.
2025-12-04T14:26:42.2840281Z 
2025-12-04T14:26:42.2840354Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2840594Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2840597Z 
2025-12-04T14:26:42.2840684Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2840745Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2840807Z ======================= 1 failed, 19 deselected in 9.58s =======================
2025-12-04T14:26:42.2840847Z Got exit code 1
2025-12-04T14:26:42.2841036Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda
2025-12-04T14:26:42.2841162Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2841353Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-356b117dd883bb98.xml
2025-12-04T14:26:42.2841409Z ============================= test session starts ==============================
2025-12-04T14:26:42.2841523Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2841564Z cachedir: .pytest_cache
2025-12-04T14:26:42.2841721Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2841767Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2841808Z configfile: pytest.ini
2025-12-04T14:26:42.2841970Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2842046Z collecting ... collected 60 items / 12 deselected / 48 selected
2025-12-04T14:26:42.2842100Z stepcurrent: skipping 12 already run items.
2025-12-04T14:26:42.2842144Z Running 8 items in this shard
2025-12-04T14:26:42.2842165Z 
2025-12-04T14:26:42.2842486Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda I1204 14:22:05.817000 392336 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 392405
2025-12-04T14:26:42.2842652Z I1204 14:22:05.818000 392336 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 392406
2025-12-04T14:26:42.2842818Z I1204 14:22:05.819000 392336 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 392407
2025-12-04T14:26:42.2842979Z I1204 14:22:05.819000 392336 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 392408
2025-12-04T14:26:42.2843266Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2843313Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2843891Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2843932Z   _warn_cpu_init()
2025-12-04T14:26:42.2844211Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2844257Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2844534Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2844579Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2844853Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2844900Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2845471Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2845510Z   _warn_cpu_init()
2025-12-04T14:26:42.2846076Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2846114Z   _warn_cpu_init()
2025-12-04T14:26:42.2846694Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2846731Z   _warn_cpu_init()
2025-12-04T14:26:42.2847065Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2847170Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2847470Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2847571Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2847857Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2847944Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2848228Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2848312Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2848607Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2848652Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2848881Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2848922Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2849146Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2849187Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2849410Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2849450Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2849670Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2849709Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2849929Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2849968Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2850188Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2850229Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2850449Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2850491Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2850710Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2850750Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2850907Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2851073Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2851374Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2851552Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2851839Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2851966Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2852244Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2852397Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2852675Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2852825Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2853105Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2853242Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2853522Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2853673Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2854169Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2854286Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2854483Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2854854Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2854970Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2855196Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2855362Z [rank2]:E1204 14:22:13.393000 392407 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2855413Z dist init r=2, world=4
2025-12-04T14:26:42.2855553Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2855724Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2856027Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2856182Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2856471Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2856598Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2856921Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2857070Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2857346Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2857493Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2857769Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2857906Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2858184Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2858335Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2858820Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2858935Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2859134Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2859514Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2859629Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2859853Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2860017Z [rank0]:E1204 14:22:13.394000 392405 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2860082Z dist init r=0, world=4
2025-12-04T14:26:42.2860223Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2860383Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2860677Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2860832Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2861117Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2861242Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2861518Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2861667Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2861943Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2862090Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2862368Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2862504Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2862784Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2862933Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2863419Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 118272 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2863535Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2863740Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2864109Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2864233Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2864471Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2864635Z [rank1]:E1204 14:22:13.398000 392406 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2864776Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2864936Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2865223Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2865379Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2865664Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2865788Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2866064Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2866214Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2866489Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2866638Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2866974Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2867109Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2867388Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2867536Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2868035Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 124416 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2868150Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2869212Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2869593Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2869720Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2869932Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2870097Z [rank3]:E1204 14:22:13.398000 392408 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2870138Z dist init r=1, world=4
2025-12-04T14:26:42.2870175Z dist init r=3, world=4
2025-12-04T14:26:42.2870512Z [rank0]:[W1204 14:22:13.075833024 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2870554Z FAILED [9.4157s] [ 12%]
2025-12-04T14:26:42.2870557Z 
2025-12-04T14:26:42.2870612Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2870718Z _ TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda _
2025-12-04T14:26:42.2870764Z Traceback (most recent call last):
2025-12-04T14:26:42.2870929Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2870972Z     self._join_processes(fn)
2025-12-04T14:26:42.2871147Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2871202Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2871380Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2871424Z     raise RuntimeError(error)
2025-12-04T14:26:42.2871506Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2871550Z Traceback (most recent call last):
2025-12-04T14:26:42.2871711Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2871752Z     getattr(self, test_name)()
2025-12-04T14:26:42.2871911Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2871945Z     fn()
2025-12-04T14:26:42.2872098Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2872137Z     method(*args, **kwargs)
2025-12-04T14:26:42.2872288Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2872328Z     method(*args, **kwargs)
2025-12-04T14:26:42.2872479Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2872516Z     with policy():
2025-12-04T14:26:42.2872667Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2872707Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2873082Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2873095Z 
2025-12-04T14:26:42.2873171Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2873431Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2873445Z 
2025-12-04T14:26:42.2873533Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2873535Z 
2025-12-04T14:26:42.2873595Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2873640Z Traceback (most recent call last):
2025-12-04T14:26:42.2873802Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2873845Z     getattr(self, test_name)()
2025-12-04T14:26:42.2874001Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2874037Z     fn()
2025-12-04T14:26:42.2874187Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2874227Z     method(*args, **kwargs)
2025-12-04T14:26:42.2874379Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2874418Z     method(*args, **kwargs)
2025-12-04T14:26:42.2874568Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2874604Z     with policy():
2025-12-04T14:26:42.2874757Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2874797Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2875154Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 118272 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2875158Z 
2025-12-04T14:26:42.2875231Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2875473Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2875475Z 
2025-12-04T14:26:42.2875561Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2875563Z 
2025-12-04T14:26:42.2875565Z 
2025-12-04T14:26:42.2875642Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2875729Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2875966Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-356b117dd883bb98.xml -
2025-12-04T14:26:42.2876028Z =========================== short test summary info ============================
2025-12-04T14:26:42.2876286Z FAILED [9.4157s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2876333Z Traceback (most recent call last):
2025-12-04T14:26:42.2876496Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2876539Z     getattr(self, test_name)()
2025-12-04T14:26:42.2876709Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2876819Z     fn()
2025-12-04T14:26:42.2876985Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2877026Z     method(*args, **kwargs)
2025-12-04T14:26:42.2877175Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2877246Z     method(*args, **kwargs)
2025-12-04T14:26:42.2877396Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2877434Z     with policy():
2025-12-04T14:26:42.2877585Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2877626Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2877988Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2877993Z 
2025-12-04T14:26:42.2878065Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2878305Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2878309Z 
2025-12-04T14:26:42.2878394Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2878396Z 
2025-12-04T14:26:42.2878455Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.2878498Z Traceback (most recent call last):
2025-12-04T14:26:42.2878661Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2878702Z     getattr(self, test_name)()
2025-12-04T14:26:42.2878863Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2878896Z     fn()
2025-12-04T14:26:42.2879047Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2879087Z     method(*args, **kwargs)
2025-12-04T14:26:42.2879238Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2879276Z     method(*args, **kwargs)
2025-12-04T14:26:42.2879425Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2879461Z     with policy():
2025-12-04T14:26:42.2879613Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2879653Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2880014Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 118272 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2880017Z 
2025-12-04T14:26:42.2880095Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2880332Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2880334Z 
2025-12-04T14:26:42.2880421Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2880499Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2880562Z ======================= 1 failed, 12 deselected in 9.58s =======================
2025-12-04T14:26:42.2880617Z Got exit code 1
2025-12-04T14:26:42.2880658Z Retrying single test...
2025-12-04T14:26:42.2880847Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-aca5d48cf6cd2d03.xml
2025-12-04T14:26:42.2880906Z ============================= test session starts ==============================
2025-12-04T14:26:42.2881043Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2881085Z cachedir: .pytest_cache
2025-12-04T14:26:42.2881241Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2881287Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2881327Z configfile: pytest.ini
2025-12-04T14:26:42.2881490Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2881565Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2881800Z stepcurrent: skipping 12 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2881845Z Running 1 items in this shard
2025-12-04T14:26:42.2881847Z 
2025-12-04T14:26:42.2882166Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda I1204 14:22:17.594000 392738 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 392807
2025-12-04T14:26:42.2882322Z I1204 14:22:17.595000 392738 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 392808
2025-12-04T14:26:42.2882474Z I1204 14:22:17.596000 392738 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 392809
2025-12-04T14:26:42.2882626Z I1204 14:22:17.596000 392738 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 392810
2025-12-04T14:26:42.2882912Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2882960Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2883537Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2883574Z   _warn_cpu_init()
2025-12-04T14:26:42.2883856Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2883902Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2884475Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2884512Z   _warn_cpu_init()
2025-12-04T14:26:42.2884810Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2884912Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2885198Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2885306Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2885599Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2885642Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2885920Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2885967Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2886540Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2886577Z   _warn_cpu_init()
2025-12-04T14:26:42.2886900Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2886944Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2887509Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2887547Z   _warn_cpu_init()
2025-12-04T14:26:42.2887833Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2887918Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2888203Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2888290Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2888522Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2888566Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2888790Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2888832Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2889067Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2889108Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2889353Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2889394Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2889626Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2889682Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2889899Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2889939Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2890159Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2890199Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2890419Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2890459Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2890605Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2890770Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2891065Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2891221Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2891513Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2891639Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2891920Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2892070Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2892348Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2892497Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2892774Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2892912Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2893205Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2893354Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2893868Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 120320 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2893996Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2894195Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2894562Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2894678Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2894892Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2895059Z [rank2]:E1204 14:22:25.207000 392809 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2895099Z dist init r=2, world=4
2025-12-04T14:26:42.2895238Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2895399Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2895687Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2895843Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2896133Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2896260Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2896541Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2896692Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2897013Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2897163Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2897442Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2897593Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2897875Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2898037Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2898551Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 124416 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2898670Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2898865Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2899233Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2899349Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2899562Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2899728Z [rank0]:E1204 14:22:25.218000 392807 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2899767Z dist init r=0, world=4
2025-12-04T14:26:42.2899906Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2900067Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2900355Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2900510Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2900797Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2900921Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2901202Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2901353Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2901630Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2901789Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2902064Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2902214Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2902594Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2902757Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2903244Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2903360Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2903556Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2903924Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2904040Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2904252Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2904417Z [rank3]:E1204 14:22:25.239000 392810 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2904456Z dist init r=3, world=4
2025-12-04T14:26:42.2904595Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2904759Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2905048Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2905203Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2905491Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2905616Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2905896Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2906045Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2906338Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2906494Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2906830Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2906979Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2907257Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2907407Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2907898Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 112128 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2908015Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2908210Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2908578Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2908691Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2908905Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2909070Z [rank1]:E1204 14:22:25.247000 392808 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2909109Z dist init r=1, world=4
2025-12-04T14:26:42.2909447Z [rank0]:[W1204 14:22:25.974662792 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2909487Z FAILED [9.7145s] [100%]
2025-12-04T14:26:42.2909489Z 
2025-12-04T14:26:42.2909547Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2909654Z _ TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda _
2025-12-04T14:26:42.2909701Z Traceback (most recent call last):
2025-12-04T14:26:42.2909867Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2909914Z     self._join_processes(fn)
2025-12-04T14:26:42.2910088Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2910143Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2910335Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2910380Z     raise RuntimeError(error)
2025-12-04T14:26:42.2910460Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2910520Z Traceback (most recent call last):
2025-12-04T14:26:42.2910682Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2910725Z     getattr(self, test_name)()
2025-12-04T14:26:42.2910895Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2910943Z     fn()
2025-12-04T14:26:42.2911094Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2911136Z     method(*args, **kwargs)
2025-12-04T14:26:42.2911286Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2911328Z     method(*args, **kwargs)
2025-12-04T14:26:42.2911477Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2911516Z     with policy():
2025-12-04T14:26:42.2911670Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2911711Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2912073Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 124416 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2912077Z 
2025-12-04T14:26:42.2912152Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2912392Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2912395Z 
2025-12-04T14:26:42.2912482Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2912485Z 
2025-12-04T14:26:42.2912545Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2912590Z Traceback (most recent call last):
2025-12-04T14:26:42.2912754Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2912796Z     getattr(self, test_name)()
2025-12-04T14:26:42.2912956Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2912989Z     fn()
2025-12-04T14:26:42.2913140Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2913181Z     method(*args, **kwargs)
2025-12-04T14:26:42.2913331Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2913372Z     method(*args, **kwargs)
2025-12-04T14:26:42.2913522Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2913559Z     with policy():
2025-12-04T14:26:42.2913709Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2913752Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2914109Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2914112Z 
2025-12-04T14:26:42.2914196Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2914434Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2914447Z 
2025-12-04T14:26:42.2914535Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2914537Z 
2025-12-04T14:26:42.2914539Z 
2025-12-04T14:26:42.2914615Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2914729Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2917127Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-aca5d48cf6cd2d03.xml -
2025-12-04T14:26:42.2917195Z =========================== short test summary info ============================
2025-12-04T14:26:42.2917454Z FAILED [9.7145s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2917503Z Traceback (most recent call last):
2025-12-04T14:26:42.2917671Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2917714Z     getattr(self, test_name)()
2025-12-04T14:26:42.2917876Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2917913Z     fn()
2025-12-04T14:26:42.2918063Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2918103Z     method(*args, **kwargs)
2025-12-04T14:26:42.2918255Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2918299Z     method(*args, **kwargs)
2025-12-04T14:26:42.2918448Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2918487Z     with policy():
2025-12-04T14:26:42.2918638Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2918679Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2919040Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 124416 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2919047Z 
2025-12-04T14:26:42.2919121Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2919362Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2919364Z 
2025-12-04T14:26:42.2919451Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2919453Z 
2025-12-04T14:26:42.2919513Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.2919557Z Traceback (most recent call last):
2025-12-04T14:26:42.2919721Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2919763Z     getattr(self, test_name)()
2025-12-04T14:26:42.2919923Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2919957Z     fn()
2025-12-04T14:26:42.2920107Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2920173Z     method(*args, **kwargs)
2025-12-04T14:26:42.2920325Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2920380Z     method(*args, **kwargs)
2025-12-04T14:26:42.2920529Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2920565Z     with policy():
2025-12-04T14:26:42.2920732Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2920795Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2921152Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 108032 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2921154Z 
2025-12-04T14:26:42.2921228Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2921465Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2921468Z 
2025-12-04T14:26:42.2921554Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2921617Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2921681Z ======================= 1 failed, 19 deselected in 9.87s =======================
2025-12-04T14:26:42.2921717Z Got exit code 1
2025-12-04T14:26:42.2921758Z Retrying single test...
2025-12-04T14:26:42.2921947Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-0ff8059988240a76.xml
2025-12-04T14:26:42.2922008Z ============================= test session starts ==============================
2025-12-04T14:26:42.2922122Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2922165Z cachedir: .pytest_cache
2025-12-04T14:26:42.2922323Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2922370Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2922411Z configfile: pytest.ini
2025-12-04T14:26:42.2922575Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2922651Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2922885Z stepcurrent: skipping 12 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2922929Z Running 1 items in this shard
2025-12-04T14:26:42.2922931Z 
2025-12-04T14:26:42.2923247Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda I1204 14:22:29.640000 393140 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 393209
2025-12-04T14:26:42.2923403Z I1204 14:22:29.640000 393140 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 393210
2025-12-04T14:26:42.2923554Z I1204 14:22:29.641000 393140 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 393211
2025-12-04T14:26:42.2923706Z I1204 14:22:29.642000 393140 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 393212
2025-12-04T14:26:42.2923993Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2924054Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2924638Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2924710Z   _warn_cpu_init()
2025-12-04T14:26:42.2925004Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2925093Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2925388Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2925431Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2925710Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2925756Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2926035Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2926080Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2926355Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:485: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2926400Z   return wrapper_cls(module, **kwargs)
2025-12-04T14:26:42.2927009Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2927048Z   _warn_cpu_init()
2025-12-04T14:26:42.2927617Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2927656Z   _warn_cpu_init()
2025-12-04T14:26:42.2928218Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2928255Z   _warn_cpu_init()
2025-12-04T14:26:42.2928557Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2928644Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2928945Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2929030Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2929343Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:532: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.2929428Z   fsdp_model = FSDP(model, auto_wrap_policy=always_wrap_policy, **fsdp_kwargs)
2025-12-04T14:26:42.2929658Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2929701Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2929926Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2929969Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2930190Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2930233Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2930453Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.2930494Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2930714Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2930755Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2930975Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2931015Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2931234Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2931275Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2931493Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.2931533Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2931680Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2931842Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2932135Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2932293Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2932580Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2932716Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2932995Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2933157Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2933445Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2933605Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2933884Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2934023Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2934302Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2934454Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2934945Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 116224 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2935060Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2935257Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2935625Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2935740Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2935955Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2936120Z [rank0]:E1204 14:22:37.288000 393209 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2936161Z dist init r=0, world=4
2025-12-04T14:26:42.2936300Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2936460Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2936786Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2936941Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2937246Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2937388Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2937684Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2937845Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2938122Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2938268Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2938547Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2938685Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2938963Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2939110Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2939597Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 120320 on device 1. CUDA driver allocated memory was 2317352960 and is now 3852468224.
2025-12-04T14:26:42.2939714Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2939912Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2940281Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2940394Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2940609Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2940774Z [rank1]:E1204 14:22:37.295000 393210 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2940814Z dist init r=1, world=4
2025-12-04T14:26:42.2940953Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2941112Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2941412Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2941577Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2941861Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2942008Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2942289Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2942438Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2942713Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2942861Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2943137Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2943274Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2943551Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2943701Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2944186Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 112128 on device 3. CUDA driver allocated memory was 2250244096 and is now 3785359360.
2025-12-04T14:26:42.2944299Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2944501Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2944867Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2944981Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2945194Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2945358Z [rank3]:E1204 14:22:37.299000 393212 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2945397Z dist init r=3, world=4
2025-12-04T14:26:42.2945546Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2945707Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2946007Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2946173Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2946467Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2946594Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2946917Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2947066Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2947344Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2947491Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2947768Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2947903Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2948182Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2948332Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2948820Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 124416 on device 2. CUDA driver allocated memory was 2300575744 and is now 3835691008.
2025-12-04T14:26:42.2948934Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2949129Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2949494Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2949607Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2949833Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2949995Z [rank2]:E1204 14:22:37.342000 393211 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2950051Z dist init r=2, world=4
2025-12-04T14:26:42.2950388Z [rank0]:[W1204 14:22:37.945389064 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2950468Z FAILED [9.5151s] [100%]
2025-12-04T14:26:42.2950470Z 
2025-12-04T14:26:42.2950527Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2950632Z _ TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda _
2025-12-04T14:26:42.2950679Z Traceback (most recent call last):
2025-12-04T14:26:42.2950842Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2950886Z     self._join_processes(fn)
2025-12-04T14:26:42.2951059Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2951115Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2951294Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2951339Z     raise RuntimeError(error)
2025-12-04T14:26:42.2951419Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2951464Z Traceback (most recent call last):
2025-12-04T14:26:42.2951624Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2951666Z     getattr(self, test_name)()
2025-12-04T14:26:42.2951824Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2951859Z     fn()
2025-12-04T14:26:42.2952011Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2952052Z     method(*args, **kwargs)
2025-12-04T14:26:42.2952201Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2952242Z     method(*args, **kwargs)
2025-12-04T14:26:42.2952392Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2952432Z     with policy():
2025-12-04T14:26:42.2952583Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2952623Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2952983Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 116224 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2952986Z 
2025-12-04T14:26:42.2953060Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2953300Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2953303Z 
2025-12-04T14:26:42.2953390Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2953392Z 
2025-12-04T14:26:42.2953394Z 
2025-12-04T14:26:42.2953470Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2953558Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2953802Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-0ff8059988240a76.xml -
2025-12-04T14:26:42.2953881Z =========================== short test summary info ============================
2025-12-04T14:26:42.2954136Z FAILED [9.5151s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2954206Z Traceback (most recent call last):
2025-12-04T14:26:42.2954370Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2954413Z     getattr(self, test_name)()
2025-12-04T14:26:42.2954571Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2954606Z     fn()
2025-12-04T14:26:42.2954757Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2954798Z     method(*args, **kwargs)
2025-12-04T14:26:42.2954947Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2954987Z     method(*args, **kwargs)
2025-12-04T14:26:42.2955136Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2955176Z     with policy():
2025-12-04T14:26:42.2955328Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2955369Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2955729Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 116224 on device 0. CUDA driver allocated memory was 2453667840 and is now 3988783104.
2025-12-04T14:26:42.2955733Z 
2025-12-04T14:26:42.2955807Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2956048Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2956050Z 
2025-12-04T14:26:42.2956136Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2956200Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2956261Z ======================= 1 failed, 19 deselected in 9.68s =======================
2025-12-04T14:26:42.2956298Z Got exit code 1
2025-12-04T14:26:42.2956485Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda
2025-12-04T14:26:42.2956614Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.2956846Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-5268b0dfa5d7f464.xml
2025-12-04T14:26:42.2956904Z ============================= test session starts ==============================
2025-12-04T14:26:42.2957016Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2957059Z cachedir: .pytest_cache
2025-12-04T14:26:42.2957218Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2957265Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2957305Z configfile: pytest.ini
2025-12-04T14:26:42.2957483Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2957559Z collecting ... collected 60 items / 13 deselected / 47 selected
2025-12-04T14:26:42.2957612Z stepcurrent: skipping 13 already run items.
2025-12-04T14:26:42.2957670Z Running 7 items in this shard
2025-12-04T14:26:42.2957672Z 
2025-12-04T14:26:42.2958040Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda I1204 14:22:41.882000 393542 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 393611
2025-12-04T14:26:42.2958229Z I1204 14:22:41.883000 393542 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 393612
2025-12-04T14:26:42.2958382Z I1204 14:22:41.884000 393542 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 393613
2025-12-04T14:26:42.2958534Z I1204 14:22:41.885000 393542 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 393614
2025-12-04T14:26:42.2959112Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2959153Z   _warn_cpu_init()
2025-12-04T14:26:42.2959720Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2959757Z   _warn_cpu_init()
2025-12-04T14:26:42.2960322Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2960359Z   _warn_cpu_init()
2025-12-04T14:26:42.2960921Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2960959Z   _warn_cpu_init()
2025-12-04T14:26:42.2961250Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2961294Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2961438Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2961600Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2961901Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2962059Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2962360Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2962495Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2962783Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2962932Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2963207Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2963356Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2963632Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2963770Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2964049Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2964197Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2964738Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 55808 on device 1. CUDA driver allocated memory was 2317352960 and is now 3458203648.
2025-12-04T14:26:42.2964856Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2965051Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2965470Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2965586Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2965799Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2965964Z [rank1]:E1204 14:22:47.663000 393612 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2966002Z dist init r=1, world=4
2025-12-04T14:26:42.2966152Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2966312Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2966613Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2966830Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2967132Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2967256Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2967533Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2967682Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2967959Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2968107Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2968384Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2968520Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2968799Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2968947Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2969484Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 64000 on device 0. CUDA driver allocated memory was 2453667840 and is now 3594518528.
2025-12-04T14:26:42.2969598Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2969794Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2970211Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2970324Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2970554Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2970719Z [rank0]:E1204 14:22:47.665000 393611 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.2970776Z dist init r=0, world=4
2025-12-04T14:26:42.2970916Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2971085Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2971386Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2971540Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2971827Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2971952Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2972229Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2972379Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2972654Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2972801Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2973078Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2973213Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2973490Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2973638Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2974175Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 61952 on device 2. CUDA driver allocated memory was 2300575744 and is now 3441426432.
2025-12-04T14:26:42.2974293Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2974490Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2974915Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2975028Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2975255Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2975430Z [rank2]:E1204 14:22:47.676000 393613 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.2975481Z dist init r=2, world=4
2025-12-04T14:26:42.2975618Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2975778Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2976065Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2976221Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2976508Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2976631Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2976951Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2977098Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2977378Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2977525Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2977803Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2977938Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2978219Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2978369Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2978902Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 64000 on device 3. CUDA driver allocated memory was 2250244096 and is now 3391094784.
2025-12-04T14:26:42.2979017Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2979225Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2979653Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2979796Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2980007Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2980171Z [rank3]:E1204 14:22:47.688000 393614 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.2980210Z dist init r=3, world=4
2025-12-04T14:26:42.2980548Z [rank0]:[W1204 14:22:47.350229837 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.2980588Z FAILED [7.4146s] [ 14%]
2025-12-04T14:26:42.2980590Z 
2025-12-04T14:26:42.2980646Z =================================== FAILURES ===================================
2025-12-04T14:26:42.2980802Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda _
2025-12-04T14:26:42.2980849Z Traceback (most recent call last):
2025-12-04T14:26:42.2981012Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.2981057Z     self._join_processes(fn)
2025-12-04T14:26:42.2981231Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.2981285Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.2981465Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.2981508Z     raise RuntimeError(error)
2025-12-04T14:26:42.2981589Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2981634Z Traceback (most recent call last):
2025-12-04T14:26:42.2981796Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2981838Z     getattr(self, test_name)()
2025-12-04T14:26:42.2981995Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2982029Z     fn()
2025-12-04T14:26:42.2982181Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2982222Z     method(*args, **kwargs)
2025-12-04T14:26:42.2982373Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2982412Z     method(*args, **kwargs)
2025-12-04T14:26:42.2982563Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2982601Z     with policy():
2025-12-04T14:26:42.2982753Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2982793Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2983214Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 64000 on device 0. CUDA driver allocated memory was 2453667840 and is now 3594518528.
2025-12-04T14:26:42.2983228Z 
2025-12-04T14:26:42.2983304Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2983591Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2983606Z 
2025-12-04T14:26:42.2983713Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2983715Z 
2025-12-04T14:26:42.2983717Z 
2025-12-04T14:26:42.2983791Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.2983880Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.2984115Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-5268b0dfa5d7f464.xml -
2025-12-04T14:26:42.2984175Z =========================== short test summary info ============================
2025-12-04T14:26:42.2984478Z FAILED [7.4146s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.2984525Z Traceback (most recent call last):
2025-12-04T14:26:42.2984689Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2984732Z     getattr(self, test_name)()
2025-12-04T14:26:42.2984892Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2984926Z     fn()
2025-12-04T14:26:42.2985078Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2985117Z     method(*args, **kwargs)
2025-12-04T14:26:42.2985269Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2985308Z     method(*args, **kwargs)
2025-12-04T14:26:42.2985457Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2985494Z     with policy():
2025-12-04T14:26:42.2985646Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2985686Z     raise RuntimeError(msg)
2025-12-04T14:26:42.2986090Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 64000 on device 0. CUDA driver allocated memory was 2453667840 and is now 3594518528.
2025-12-04T14:26:42.2986092Z 
2025-12-04T14:26:42.2986167Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2986454Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2986456Z 
2025-12-04T14:26:42.2986546Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2986609Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.2986670Z ======================= 1 failed, 13 deselected in 7.57s =======================
2025-12-04T14:26:42.2986706Z Got exit code 1
2025-12-04T14:26:42.2986790Z Retrying single test...
2025-12-04T14:26:42.2986994Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-ef1c390d25d8c8ab.xml
2025-12-04T14:26:42.2987052Z ============================= test session starts ==============================
2025-12-04T14:26:42.2987179Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.2987221Z cachedir: .pytest_cache
2025-12-04T14:26:42.2987378Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.2987453Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.2987494Z configfile: pytest.ini
2025-12-04T14:26:42.2987655Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.2987730Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.2988013Z stepcurrent: skipping 13 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2988057Z Running 1 items in this shard
2025-12-04T14:26:42.2988060Z 
2025-12-04T14:26:42.2988422Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda I1204 14:22:51.712000 393944 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 394013
2025-12-04T14:26:42.2988580Z I1204 14:22:51.712000 393944 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 394014
2025-12-04T14:26:42.2988731Z I1204 14:22:51.713000 393944 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 394015
2025-12-04T14:26:42.2988882Z I1204 14:22:51.714000 393944 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 394016
2025-12-04T14:26:42.2989462Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2989501Z   _warn_cpu_init()
2025-12-04T14:26:42.2990070Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2990109Z   _warn_cpu_init()
2025-12-04T14:26:42.2990677Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2990716Z   _warn_cpu_init()
2025-12-04T14:26:42.2991291Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.2991327Z   _warn_cpu_init()
2025-12-04T14:26:42.2991630Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.2991672Z   return func(*args, **kwargs)
2025-12-04T14:26:42.2991825Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2991998Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2992287Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2992443Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2992729Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2992853Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2993136Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2993285Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2993563Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2993711Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2993988Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2994124Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2994402Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2994549Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2995085Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3458203648.
2025-12-04T14:26:42.2995202Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2995398Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.2995825Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.2995949Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.2996172Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.2996347Z [rank1]:E1204 14:22:57.425000 394014 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.2996385Z dist init r=1, world=4
2025-12-04T14:26:42.2996523Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.2996684Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.2997017Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.2997170Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.2997458Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.2997582Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.2997864Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2998013Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2998289Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.2998437Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.2998712Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.2998848Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.2999126Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.2999274Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.2999807Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 57856 on device 2. CUDA driver allocated memory was 2300575744 and is now 3441426432.
2025-12-04T14:26:42.2999936Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3000147Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3000580Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3000708Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3000918Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3001083Z [rank2]:E1204 14:22:57.441000 394015 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3001123Z dist init r=2, world=4
2025-12-04T14:26:42.3001259Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3001420Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3001710Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3001864Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3002152Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3002278Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3002557Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3002706Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3002983Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3003130Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3003408Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3003544Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3003825Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3003973Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3004516Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 66048 on device 0. CUDA driver allocated memory was 2453667840 and is now 3594518528.
2025-12-04T14:26:42.3004645Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3004861Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3005276Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3005389Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3005602Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3005766Z [rank0]:E1204 14:22:57.443000 394013 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3005806Z dist init r=0, world=4
2025-12-04T14:26:42.3005943Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3006102Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3006390Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3006543Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3006874Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3006998Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3007275Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3007423Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3007699Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3007845Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3008120Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3008257Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3008547Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3008711Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3009255Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 3. CUDA driver allocated memory was 2250244096 and is now 3391094784.
2025-12-04T14:26:42.3009383Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3009579Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3009993Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3010107Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3010318Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3010482Z [rank3]:E1204 14:22:57.451000 394016 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3010520Z dist init r=3, world=4
2025-12-04T14:26:42.3010856Z [rank0]:[W1204 14:22:57.175218595 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3010896Z FAILED [7.4158s] [100%]
2025-12-04T14:26:42.3010898Z 
2025-12-04T14:26:42.3010952Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3011107Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda _
2025-12-04T14:26:42.3011153Z Traceback (most recent call last):
2025-12-04T14:26:42.3011316Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3011358Z     self._join_processes(fn)
2025-12-04T14:26:42.3011533Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3011586Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3011764Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3011806Z     raise RuntimeError(error)
2025-12-04T14:26:42.3011887Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3011931Z Traceback (most recent call last):
2025-12-04T14:26:42.3012094Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3012136Z     getattr(self, test_name)()
2025-12-04T14:26:42.3012293Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3012327Z     fn()
2025-12-04T14:26:42.3012492Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3012534Z     method(*args, **kwargs)
2025-12-04T14:26:42.3012699Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3012739Z     method(*args, **kwargs)
2025-12-04T14:26:42.3012887Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3012935Z     with policy():
2025-12-04T14:26:42.3013101Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3013143Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3013547Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3458203648.
2025-12-04T14:26:42.3013550Z 
2025-12-04T14:26:42.3013625Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3013912Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3013914Z 
2025-12-04T14:26:42.3014005Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3014007Z 
2025-12-04T14:26:42.3014068Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.3014112Z Traceback (most recent call last):
2025-12-04T14:26:42.3014274Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3014315Z     getattr(self, test_name)()
2025-12-04T14:26:42.3014475Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3014510Z     fn()
2025-12-04T14:26:42.3014661Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3014699Z     method(*args, **kwargs)
2025-12-04T14:26:42.3014850Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3014890Z     method(*args, **kwargs)
2025-12-04T14:26:42.3015040Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3015076Z     with policy():
2025-12-04T14:26:42.3015226Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3015266Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3015668Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 57856 on device 2. CUDA driver allocated memory was 2300575744 and is now 3441426432.
2025-12-04T14:26:42.3015671Z 
2025-12-04T14:26:42.3015744Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3016032Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3016035Z 
2025-12-04T14:26:42.3016122Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3016124Z 
2025-12-04T14:26:42.3016128Z 
2025-12-04T14:26:42.3016202Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3016301Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3016535Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-ef1c390d25d8c8ab.xml -
2025-12-04T14:26:42.3016618Z =========================== short test summary info ============================
2025-12-04T14:26:42.3016972Z FAILED [7.4158s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3017036Z Traceback (most recent call last):
2025-12-04T14:26:42.3017200Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3017242Z     getattr(self, test_name)()
2025-12-04T14:26:42.3017400Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3017435Z     fn()
2025-12-04T14:26:42.3017584Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3017625Z     method(*args, **kwargs)
2025-12-04T14:26:42.3017776Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3017817Z     method(*args, **kwargs)
2025-12-04T14:26:42.3017967Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3018004Z     with policy():
2025-12-04T14:26:42.3018155Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3018195Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3018599Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 1. CUDA driver allocated memory was 2317352960 and is now 3458203648.
2025-12-04T14:26:42.3018602Z 
2025-12-04T14:26:42.3018674Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3018963Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3018966Z 
2025-12-04T14:26:42.3019052Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3019054Z 
2025-12-04T14:26:42.3019112Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.3019157Z Traceback (most recent call last):
2025-12-04T14:26:42.3019319Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3019362Z     getattr(self, test_name)()
2025-12-04T14:26:42.3019522Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3019556Z     fn()
2025-12-04T14:26:42.3019705Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3019746Z     method(*args, **kwargs)
2025-12-04T14:26:42.3019895Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3019935Z     method(*args, **kwargs)
2025-12-04T14:26:42.3020083Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3020120Z     with policy():
2025-12-04T14:26:42.3020282Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3020338Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3020739Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 57856 on device 2. CUDA driver allocated memory was 2300575744 and is now 3441426432.
2025-12-04T14:26:42.3020753Z 
2025-12-04T14:26:42.3020837Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3021121Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3021124Z 
2025-12-04T14:26:42.3021211Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3021274Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3021335Z ======================= 1 failed, 19 deselected in 7.56s =======================
2025-12-04T14:26:42.3021373Z Got exit code 1
2025-12-04T14:26:42.3021412Z Retrying single test...
2025-12-04T14:26:42.3021601Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-c1a3c6146324a002.xml
2025-12-04T14:26:42.3021662Z ============================= test session starts ==============================
2025-12-04T14:26:42.3021774Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3021815Z cachedir: .pytest_cache
2025-12-04T14:26:42.3021972Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3022018Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3022060Z configfile: pytest.ini
2025-12-04T14:26:42.3022220Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3022297Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3022578Z stepcurrent: skipping 13 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3022625Z Running 1 items in this shard
2025-12-04T14:26:42.3022627Z 
2025-12-04T14:26:42.3022992Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda I1204 14:23:01.445000 394346 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 394415
2025-12-04T14:26:42.3023147Z I1204 14:23:01.446000 394346 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 394416
2025-12-04T14:26:42.3023299Z I1204 14:23:01.447000 394346 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 394417
2025-12-04T14:26:42.3023450Z I1204 14:23:01.447000 394346 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 394418
2025-12-04T14:26:42.3024029Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3024067Z   _warn_cpu_init()
2025-12-04T14:26:42.3024644Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3024692Z   _warn_cpu_init()
2025-12-04T14:26:42.3025281Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3025320Z   _warn_cpu_init()
2025-12-04T14:26:42.3025888Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3025926Z   _warn_cpu_init()
2025-12-04T14:26:42.3026219Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3026262Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3026407Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3026570Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3026900Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3027056Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3027342Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3027465Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3027745Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3027894Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3028170Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3028317Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3028606Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3028743Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3029034Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3029196Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3029744Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3594518528.
2025-12-04T14:26:42.3029859Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3030057Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3030474Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3030591Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3030805Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3030969Z [rank0]:E1204 14:23:07.263000 394415 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3031009Z dist init r=0, world=4
2025-12-04T14:26:42.3031149Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3031308Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3031597Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3031752Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3032038Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3032165Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3032449Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3032600Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3032895Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3033042Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3033329Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3033474Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3033764Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3033912Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3034442Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 53760 on device 1. CUDA driver allocated memory was 2317352960 and is now 3458203648.
2025-12-04T14:26:42.3034557Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3034753Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3035168Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3035282Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3035494Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3035660Z [rank1]:E1204 14:23:07.267000 394416 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3035699Z dist init r=1, world=4
2025-12-04T14:26:42.3035840Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3035998Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3036285Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3036439Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3036727Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3036883Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3037177Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3037325Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3037620Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3037779Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3038068Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3038204Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3038481Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3038632Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3039164Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 59904 on device 3. CUDA driver allocated memory was 2250244096 and is now 3391094784.
2025-12-04T14:26:42.3039279Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3039477Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3039894Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3040010Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3040222Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3040384Z [rank3]:E1204 14:23:07.287000 394418 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3040424Z dist init r=3, world=4
2025-12-04T14:26:42.3040560Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3040720Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3041008Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3041164Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3041458Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3041584Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3041871Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3042019Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3042323Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3042470Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3042748Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3042884Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3043163Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3043310Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3043839Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 59904 on device 2. CUDA driver allocated memory was 2300575744 and is now 3441426432.
2025-12-04T14:26:42.3043953Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3044147Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3044565Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3044677Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3044888Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3045053Z [rank2]:E1204 14:23:07.355000 394417 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3045090Z dist init r=2, world=4
2025-12-04T14:26:42.3045430Z [rank0]:[W1204 14:23:07.973542294 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3045470Z FAILED [7.5133s] [100%]
2025-12-04T14:26:42.3045472Z 
2025-12-04T14:26:42.3045527Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3045690Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda _
2025-12-04T14:26:42.3045739Z Traceback (most recent call last):
2025-12-04T14:26:42.3045901Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3045955Z     self._join_processes(fn)
2025-12-04T14:26:42.3046127Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3046192Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3046387Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3046431Z     raise RuntimeError(error)
2025-12-04T14:26:42.3046511Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3046557Z Traceback (most recent call last):
2025-12-04T14:26:42.3046718Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3046804Z     getattr(self, test_name)()
2025-12-04T14:26:42.3046964Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3046998Z     fn()
2025-12-04T14:26:42.3047149Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3047190Z     method(*args, **kwargs)
2025-12-04T14:26:42.3047341Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3047382Z     method(*args, **kwargs)
2025-12-04T14:26:42.3047533Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3047569Z     with policy():
2025-12-04T14:26:42.3047721Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3047763Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3048169Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3594518528.
2025-12-04T14:26:42.3048173Z 
2025-12-04T14:26:42.3048248Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3048535Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3048537Z 
2025-12-04T14:26:42.3048626Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3048628Z 
2025-12-04T14:26:42.3048686Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3048731Z Traceback (most recent call last):
2025-12-04T14:26:42.3048895Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3048938Z     getattr(self, test_name)()
2025-12-04T14:26:42.3049095Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3049133Z     fn()
2025-12-04T14:26:42.3049283Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3049322Z     method(*args, **kwargs)
2025-12-04T14:26:42.3049470Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3049509Z     method(*args, **kwargs)
2025-12-04T14:26:42.3049673Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3049727Z     with policy():
2025-12-04T14:26:42.3049878Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3049919Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3050336Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 53760 on device 1. CUDA driver allocated memory was 2317352960 and is now 3458203648.
2025-12-04T14:26:42.3050353Z 
2025-12-04T14:26:42.3050427Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3050714Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3050717Z 
2025-12-04T14:26:42.3050804Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3050806Z 
2025-12-04T14:26:42.3050808Z 
2025-12-04T14:26:42.3050882Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3050969Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3051205Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-c1a3c6146324a002.xml -
2025-12-04T14:26:42.3051265Z =========================== short test summary info ============================
2025-12-04T14:26:42.3051567Z FAILED [7.5133s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3051612Z Traceback (most recent call last):
2025-12-04T14:26:42.3051777Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3051819Z     getattr(self, test_name)()
2025-12-04T14:26:42.3051979Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3052016Z     fn()
2025-12-04T14:26:42.3052168Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3052207Z     method(*args, **kwargs)
2025-12-04T14:26:42.3052358Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3052397Z     method(*args, **kwargs)
2025-12-04T14:26:42.3052546Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3052583Z     with policy():
2025-12-04T14:26:42.3052735Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3052776Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3053181Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 49664 on device 0. CUDA driver allocated memory was 2453667840 and is now 3594518528.
2025-12-04T14:26:42.3053184Z 
2025-12-04T14:26:42.3053257Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3053557Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3053561Z 
2025-12-04T14:26:42.3053646Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3053659Z 
2025-12-04T14:26:42.3053718Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3053762Z Traceback (most recent call last):
2025-12-04T14:26:42.3053924Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3053987Z     getattr(self, test_name)()
2025-12-04T14:26:42.3054146Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3054179Z     fn()
2025-12-04T14:26:42.3054329Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3054367Z     method(*args, **kwargs)
2025-12-04T14:26:42.3054518Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3054557Z     method(*args, **kwargs)
2025-12-04T14:26:42.3054707Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3054743Z     with policy():
2025-12-04T14:26:42.3054895Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3054936Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3055337Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda! Caching allocator allocated memory was 512 and is now reported as 53760 on device 1. CUDA driver allocated memory was 2317352960 and is now 3458203648.
2025-12-04T14:26:42.3055339Z 
2025-12-04T14:26:42.3055412Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3055698Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3055701Z 
2025-12-04T14:26:42.3055787Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3055849Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3055913Z ======================= 1 failed, 19 deselected in 7.68s =======================
2025-12-04T14:26:42.3055949Z Got exit code 1
2025-12-04T14:26:42.3056183Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda
2025-12-04T14:26:42.3056310Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.3056501Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4feadb560030530d.xml
2025-12-04T14:26:42.3056559Z ============================= test session starts ==============================
2025-12-04T14:26:42.3056672Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3056714Z cachedir: .pytest_cache
2025-12-04T14:26:42.3056922Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3056967Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3057008Z configfile: pytest.ini
2025-12-04T14:26:42.3057169Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3057260Z collecting ... collected 60 items / 14 deselected / 46 selected
2025-12-04T14:26:42.3057314Z stepcurrent: skipping 14 already run items.
2025-12-04T14:26:42.3057358Z Running 6 items in this shard
2025-12-04T14:26:42.3057373Z 
2025-12-04T14:26:42.3057731Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda I1204 14:23:11.570000 394748 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 394817
2025-12-04T14:26:42.3057901Z I1204 14:23:11.571000 394748 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 394818
2025-12-04T14:26:42.3058067Z I1204 14:23:11.571000 394748 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 394819
2025-12-04T14:26:42.3058219Z I1204 14:23:11.572000 394748 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 394820
2025-12-04T14:26:42.3058514Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3058565Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3059150Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3059190Z   _warn_cpu_init()
2025-12-04T14:26:42.3059480Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3059529Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3060098Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3060137Z   _warn_cpu_init()
2025-12-04T14:26:42.3060424Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3060502Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3060788Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3060865Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3061150Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3061198Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3061780Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3061829Z   _warn_cpu_init()
2025-12-04T14:26:42.3062116Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3062189Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3062501Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3062552Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3063121Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3063160Z   _warn_cpu_init()
2025-12-04T14:26:42.3063444Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3063519Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3063809Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3063852Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3064081Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3064124Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3064352Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3064393Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3064616Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3064656Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3064878Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3064918Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3065138Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3065180Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3065399Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3065440Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3065659Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3065698Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3065928Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3065969Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3066127Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3066291Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3066594Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3066795Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3067081Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3067208Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3067484Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3067635Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3067911Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3068058Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3068334Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3068472Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3068749Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3068898Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3069427Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 77312 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3069544Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3069739Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3070150Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3070278Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3070492Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3070674Z [rank1]:E1204 14:23:17.396000 394818 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3070727Z dist init r=1, world=4
2025-12-04T14:26:42.3070877Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3071037Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3071327Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3071483Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3071770Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3071896Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3072173Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3072322Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3072597Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3072745Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3073021Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3073159Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3073438Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3073587Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3074116Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 73216 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3074231Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3074439Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3074844Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3074969Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3075205Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3075370Z [rank3]:E1204 14:23:17.407000 394820 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3075410Z dist init r=3, world=4
2025-12-04T14:26:42.3075549Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3075708Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3075998Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3076154Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3076437Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3076561Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3076885Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3077034Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3077312Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3077458Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3077737Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3077873Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3078151Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3078302Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3078844Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 67072 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3078961Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3079174Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3079592Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3079720Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3079932Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3080094Z [rank2]:E1204 14:23:17.420000 394819 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3080233Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3080392Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3080682Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3080836Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3081120Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3081244Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3081521Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3081669Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3081944Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3082091Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3082367Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3082502Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3082780Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3082927Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3083462Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 75264 on device 0. CUDA driver allocated memory was 2453667840 and is now 3571449856.
2025-12-04T14:26:42.3083587Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3083807Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3084216Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3084327Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3084538Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3084700Z [rank0]:E1204 14:23:17.420000 394817 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3084741Z dist init r=2, world=4
2025-12-04T14:26:42.3084778Z dist init r=0, world=4
2025-12-04T14:26:42.3085114Z [rank0]:[W1204 14:23:17.155227437 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3085154Z FAILED [7.5120s] [ 16%]
2025-12-04T14:26:42.3085156Z 
2025-12-04T14:26:42.3085213Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3085357Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda _
2025-12-04T14:26:42.3085402Z Traceback (most recent call last):
2025-12-04T14:26:42.3085564Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3085607Z     self._join_processes(fn)
2025-12-04T14:26:42.3085780Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3085833Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3086011Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3086055Z     raise RuntimeError(error)
2025-12-04T14:26:42.3086135Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3086179Z Traceback (most recent call last):
2025-12-04T14:26:42.3086341Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3086383Z     getattr(self, test_name)()
2025-12-04T14:26:42.3086541Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3086577Z     fn()
2025-12-04T14:26:42.3086729Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3086803Z     method(*args, **kwargs)
2025-12-04T14:26:42.3086954Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3086993Z     method(*args, **kwargs)
2025-12-04T14:26:42.3087159Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3087212Z     with policy():
2025-12-04T14:26:42.3087362Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3087403Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3087813Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 77312 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3087833Z 
2025-12-04T14:26:42.3087910Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3088191Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3088194Z 
2025-12-04T14:26:42.3088282Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3088284Z 
2025-12-04T14:26:42.3088343Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3088389Z Traceback (most recent call last):
2025-12-04T14:26:42.3088551Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3088595Z     getattr(self, test_name)()
2025-12-04T14:26:42.3088754Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3088788Z     fn()
2025-12-04T14:26:42.3088939Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3088979Z     method(*args, **kwargs)
2025-12-04T14:26:42.3089129Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3089168Z     method(*args, **kwargs)
2025-12-04T14:26:42.3089317Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3089353Z     with policy():
2025-12-04T14:26:42.3089505Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3089546Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3089943Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 73216 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3089945Z 
2025-12-04T14:26:42.3090019Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3090297Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3090301Z 
2025-12-04T14:26:42.3090388Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3090390Z 
2025-12-04T14:26:42.3090393Z 
2025-12-04T14:26:42.3090468Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3090556Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3090793Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-4feadb560030530d.xml -
2025-12-04T14:26:42.3090866Z =========================== short test summary info ============================
2025-12-04T14:26:42.3091159Z FAILED [7.5120s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3091217Z Traceback (most recent call last):
2025-12-04T14:26:42.3091379Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3091433Z     getattr(self, test_name)()
2025-12-04T14:26:42.3091602Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3091638Z     fn()
2025-12-04T14:26:42.3091787Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3091827Z     method(*args, **kwargs)
2025-12-04T14:26:42.3091976Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3092016Z     method(*args, **kwargs)
2025-12-04T14:26:42.3092166Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3092201Z     with policy():
2025-12-04T14:26:42.3092352Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3092393Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3092789Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 77312 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3092791Z 
2025-12-04T14:26:42.3092864Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3093142Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3093145Z 
2025-12-04T14:26:42.3093231Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3093233Z 
2025-12-04T14:26:42.3093293Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3093340Z Traceback (most recent call last):
2025-12-04T14:26:42.3093502Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3093542Z     getattr(self, test_name)()
2025-12-04T14:26:42.3093701Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3093736Z     fn()
2025-12-04T14:26:42.3093885Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3093926Z     method(*args, **kwargs)
2025-12-04T14:26:42.3094080Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3094120Z     method(*args, **kwargs)
2025-12-04T14:26:42.3094269Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3094308Z     with policy():
2025-12-04T14:26:42.3094458Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3094498Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3094911Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 73216 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3094926Z 
2025-12-04T14:26:42.3094999Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3095275Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3095290Z 
2025-12-04T14:26:42.3095387Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3095451Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3095512Z ======================= 1 failed, 14 deselected in 7.65s =======================
2025-12-04T14:26:42.3095548Z Got exit code 1
2025-12-04T14:26:42.3095588Z Retrying single test...
2025-12-04T14:26:42.3095779Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-c5fc2b98a4e5190d.xml
2025-12-04T14:26:42.3095836Z ============================= test session starts ==============================
2025-12-04T14:26:42.3095948Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3095989Z cachedir: .pytest_cache
2025-12-04T14:26:42.3096146Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3096193Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3096233Z configfile: pytest.ini
2025-12-04T14:26:42.3096395Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3096468Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3096778Z stepcurrent: skipping 14 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3096824Z Running 1 items in this shard
2025-12-04T14:26:42.3096826Z 
2025-12-04T14:26:42.3097179Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda I1204 14:23:21.661000 395150 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 395219
2025-12-04T14:26:42.3097339Z I1204 14:23:21.661000 395150 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 395220
2025-12-04T14:26:42.3097492Z I1204 14:23:21.662000 395150 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 395221
2025-12-04T14:26:42.3097641Z I1204 14:23:21.663000 395150 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 395222
2025-12-04T14:26:42.3097932Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3097983Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3098564Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3098602Z   _warn_cpu_init()
2025-12-04T14:26:42.3098905Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3098956Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3099550Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3099601Z   _warn_cpu_init()
2025-12-04T14:26:42.3099893Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3099971Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3100257Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3100333Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3100616Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3100665Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3101233Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3101271Z   _warn_cpu_init()
2025-12-04T14:26:42.3101556Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3101632Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3101917Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3101965Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3102536Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3102574Z   _warn_cpu_init()
2025-12-04T14:26:42.3102860Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3102934Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3103236Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3103280Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3103508Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3103563Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3103796Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3103849Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3104069Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3104110Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3104331Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3104372Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3104592Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3104632Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3104853Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3104893Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3105111Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3105151Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3105372Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3105413Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3105558Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3105720Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3106012Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3106169Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3106454Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3106580Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3106898Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3107049Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3107328Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3107489Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3107779Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3107916Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3108229Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3108377Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3108905Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 75264 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3109021Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3109220Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3109631Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3109745Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3109958Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3110121Z [rank1]:E1204 14:23:27.453000 395220 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3110163Z dist init r=1, world=4
2025-12-04T14:26:42.3110300Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3110462Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3110748Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3110904Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3111190Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3111316Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3111592Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3111751Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3112031Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3112188Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3112486Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3112624Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3112901Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3113051Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3113576Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 71168 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3113692Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3113888Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3114300Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3114418Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3114632Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3114797Z [rank3]:E1204 14:23:27.468000 395222 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3114835Z dist init r=3, world=4
2025-12-04T14:26:42.3114974Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3115131Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3115419Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3115575Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3115858Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3115994Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3116270Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3116430Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3116731Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3116933Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3117210Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3117346Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3117624Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3117773Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3118296Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 58880 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3118410Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3118606Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3119015Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3119129Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3119342Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3119506Z [rank2]:E1204 14:23:27.477000 395221 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3119545Z dist init r=2, world=4
2025-12-04T14:26:42.3119682Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3119845Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3120131Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3120299Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3120586Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3120722Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3121013Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3121175Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3121454Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3121601Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3121880Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3122018Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3122294Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3122443Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3122965Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 69120 on device 0. CUDA driver allocated memory was 2453667840 and is now 3571449856.
2025-12-04T14:26:42.3123081Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3123276Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3123686Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3123800Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3124010Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3124176Z [rank0]:E1204 14:23:27.485000 395219 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3124214Z dist init r=0, world=4
2025-12-04T14:26:42.3124560Z [rank0]:[W1204 14:23:27.233765743 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3124599Z FAILED [7.4137s] [100%]
2025-12-04T14:26:42.3124601Z 
2025-12-04T14:26:42.3124658Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3124812Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda _
2025-12-04T14:26:42.3124859Z Traceback (most recent call last):
2025-12-04T14:26:42.3125022Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3125093Z     self._join_processes(fn)
2025-12-04T14:26:42.3125267Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3125320Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3125499Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3125543Z     raise RuntimeError(error)
2025-12-04T14:26:42.3125624Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3125669Z Traceback (most recent call last):
2025-12-04T14:26:42.3125831Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3125872Z     getattr(self, test_name)()
2025-12-04T14:26:42.3126032Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3126067Z     fn()
2025-12-04T14:26:42.3126218Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3126258Z     method(*args, **kwargs)
2025-12-04T14:26:42.3126408Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3126448Z     method(*args, **kwargs)
2025-12-04T14:26:42.3126598Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3126635Z     with policy():
2025-12-04T14:26:42.3126831Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3126872Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3127273Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 75264 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3127276Z 
2025-12-04T14:26:42.3127352Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3127631Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3127634Z 
2025-12-04T14:26:42.3127722Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3127723Z 
2025-12-04T14:26:42.3127725Z 
2025-12-04T14:26:42.3127799Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3127887Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3128123Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-c5fc2b98a4e5190d.xml -
2025-12-04T14:26:42.3128183Z =========================== short test summary info ============================
2025-12-04T14:26:42.3128493Z FAILED [7.4137s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3128540Z Traceback (most recent call last):
2025-12-04T14:26:42.3128721Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3128763Z     getattr(self, test_name)()
2025-12-04T14:26:42.3128922Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3128984Z     fn()
2025-12-04T14:26:42.3129136Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3129175Z     method(*args, **kwargs)
2025-12-04T14:26:42.3129325Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3129365Z     method(*args, **kwargs)
2025-12-04T14:26:42.3129515Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3129552Z     with policy():
2025-12-04T14:26:42.3129705Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3129745Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3130145Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 75264 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3130148Z 
2025-12-04T14:26:42.3130221Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3130503Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3130505Z 
2025-12-04T14:26:42.3130593Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3130655Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3130716Z ======================= 1 failed, 19 deselected in 7.57s =======================
2025-12-04T14:26:42.3130752Z Got exit code 1
2025-12-04T14:26:42.3130793Z Retrying single test...
2025-12-04T14:26:42.3130983Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1a80a690a1a8dd72.xml
2025-12-04T14:26:42.3131039Z ============================= test session starts ==============================
2025-12-04T14:26:42.3131150Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3131191Z cachedir: .pytest_cache
2025-12-04T14:26:42.3131349Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3131397Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3131437Z configfile: pytest.ini
2025-12-04T14:26:42.3131600Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3131674Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3131950Z stepcurrent: skipping 14 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3131994Z Running 1 items in this shard
2025-12-04T14:26:42.3131996Z 
2025-12-04T14:26:42.3132362Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda I1204 14:23:31.470000 395552 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 395621
2025-12-04T14:26:42.3132529Z I1204 14:23:31.471000 395552 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 395622
2025-12-04T14:26:42.3132680Z I1204 14:23:31.472000 395552 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 395623
2025-12-04T14:26:42.3132842Z I1204 14:23:31.473000 395552 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 395624
2025-12-04T14:26:42.3133144Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3133196Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3133773Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3133812Z   _warn_cpu_init()
2025-12-04T14:26:42.3134100Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3134150Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3134718Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3134755Z   _warn_cpu_init()
2025-12-04T14:26:42.3135041Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3135121Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3135409Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3135483Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3135774Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3135817Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3136101Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3136152Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3136733Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3136816Z   _warn_cpu_init()
2025-12-04T14:26:42.3137117Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3137191Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3137505Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:426: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3137553Z   return FSDP(layer, group, **fsdp_kwargs)
2025-12-04T14:26:42.3138125Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3138162Z   _warn_cpu_init()
2025-12-04T14:26:42.3138449Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:1464: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3138522Z   fsdp_model = FSDP(fsdp_model, self.process_group, **fsdp_kwargs)
2025-12-04T14:26:42.3138752Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3138795Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3139019Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3139062Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3139282Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3139323Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3139544Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3139586Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3139804Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3139845Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3140064Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3140106Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3140323Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3140366Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3140584Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3140626Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3140769Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3140947Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3143041Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3143228Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3143547Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3143674Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3143954Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3144105Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3144386Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3144535Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3144812Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3144950Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3145233Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3145381Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3145915Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 71168 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3146036Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3146233Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3146650Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3146810Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3147025Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3147206Z [rank1]:E1204 14:23:37.295000 395622 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3147362Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3147520Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3147823Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3147992Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3148279Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3148403Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3148680Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3148830Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3149109Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3149257Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3149535Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3149672Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3149951Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3150101Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3150625Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 67072 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3150740Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3150936Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3151347Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3151476Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3151688Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3151862Z [rank2]:E1204 14:23:37.296000 395623 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3151902Z dist init r=1, world=4
2025-12-04T14:26:42.3151953Z dist init r=2, world=4
2025-12-04T14:26:42.3152105Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3152264Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3152552Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3152706Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3152991Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3153117Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3153393Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3153543Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3153824Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3153971Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3154248Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3154384Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3154662Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3154810Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3155334Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 67072 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3155449Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3155654Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3156064Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3156187Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3156421Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3156584Z [rank3]:E1204 14:23:37.301000 395624 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3156623Z dist init r=3, world=4
2025-12-04T14:26:42.3156809Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3156969Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3157256Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3157410Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3157695Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3157818Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3158095Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3158243Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3158522Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3158670Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3158945Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3159081Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3159358Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3159508Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3160054Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 75264 on device 0. CUDA driver allocated memory was 2453667840 and is now 3571449856.
2025-12-04T14:26:42.3160169Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3160379Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3160798Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3160926Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3161137Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3161300Z [rank0]:E1204 14:23:37.339000 395621 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3161339Z dist init r=0, world=4
2025-12-04T14:26:42.3161676Z [rank0]:[W1204 14:23:37.084948220 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3161716Z FAILED [7.5135s] [100%]
2025-12-04T14:26:42.3161719Z 
2025-12-04T14:26:42.3161775Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3161921Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda _
2025-12-04T14:26:42.3161967Z Traceback (most recent call last):
2025-12-04T14:26:42.3162132Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3162177Z     self._join_processes(fn)
2025-12-04T14:26:42.3162350Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3162403Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3162582Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3162625Z     raise RuntimeError(error)
2025-12-04T14:26:42.3162706Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3162750Z Traceback (most recent call last):
2025-12-04T14:26:42.3162911Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3162954Z     getattr(self, test_name)()
2025-12-04T14:26:42.3163111Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3163147Z     fn()
2025-12-04T14:26:42.3163297Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3163338Z     method(*args, **kwargs)
2025-12-04T14:26:42.3163488Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3163529Z     method(*args, **kwargs)
2025-12-04T14:26:42.3163678Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3163714Z     with policy():
2025-12-04T14:26:42.3163865Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3163918Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3164323Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 71168 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3164337Z 
2025-12-04T14:26:42.3164414Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3164716Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3164718Z 
2025-12-04T14:26:42.3164808Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3164810Z 
2025-12-04T14:26:42.3164869Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.3164916Z Traceback (most recent call last):
2025-12-04T14:26:42.3165078Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3165121Z     getattr(self, test_name)()
2025-12-04T14:26:42.3165279Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3165315Z     fn()
2025-12-04T14:26:42.3165465Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3165505Z     method(*args, **kwargs)
2025-12-04T14:26:42.3165654Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3165693Z     method(*args, **kwargs)
2025-12-04T14:26:42.3165842Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3165878Z     with policy():
2025-12-04T14:26:42.3166029Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3166070Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3166472Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 67072 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3166475Z 
2025-12-04T14:26:42.3166547Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3166869Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3166872Z 
2025-12-04T14:26:42.3166959Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3166962Z 
2025-12-04T14:26:42.3167021Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3167066Z Traceback (most recent call last):
2025-12-04T14:26:42.3167228Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3167271Z     getattr(self, test_name)()
2025-12-04T14:26:42.3167431Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3167466Z     fn()
2025-12-04T14:26:42.3167615Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3167656Z     method(*args, **kwargs)
2025-12-04T14:26:42.3167819Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3167859Z     method(*args, **kwargs)
2025-12-04T14:26:42.3168008Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3168059Z     with policy():
2025-12-04T14:26:42.3168209Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3168250Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3168672Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 67072 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3168675Z 
2025-12-04T14:26:42.3168748Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3169028Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3169032Z 
2025-12-04T14:26:42.3169118Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3169120Z 
2025-12-04T14:26:42.3169122Z 
2025-12-04T14:26:42.3169198Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3169287Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3169522Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1a80a690a1a8dd72.xml -
2025-12-04T14:26:42.3169582Z =========================== short test summary info ============================
2025-12-04T14:26:42.3169878Z FAILED [7.5135s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3169924Z Traceback (most recent call last):
2025-12-04T14:26:42.3170090Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3170131Z     getattr(self, test_name)()
2025-12-04T14:26:42.3170293Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3170327Z     fn()
2025-12-04T14:26:42.3170478Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3170516Z     method(*args, **kwargs)
2025-12-04T14:26:42.3170667Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3170707Z     method(*args, **kwargs)
2025-12-04T14:26:42.3170857Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3170894Z     with policy():
2025-12-04T14:26:42.3171045Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3171085Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3171482Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 71168 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3171485Z 
2025-12-04T14:26:42.3171558Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3171847Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3171862Z 
2025-12-04T14:26:42.3171949Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3171951Z 
2025-12-04T14:26:42.3172009Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.3172055Z Traceback (most recent call last):
2025-12-04T14:26:42.3172226Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3172287Z     getattr(self, test_name)()
2025-12-04T14:26:42.3172446Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3172479Z     fn()
2025-12-04T14:26:42.3172630Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3172669Z     method(*args, **kwargs)
2025-12-04T14:26:42.3172818Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3172857Z     method(*args, **kwargs)
2025-12-04T14:26:42.3173006Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3173042Z     with policy():
2025-12-04T14:26:42.3173195Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3173235Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3173631Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 67072 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3173633Z 
2025-12-04T14:26:42.3173704Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3173981Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3173984Z 
2025-12-04T14:26:42.3174070Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3174073Z 
2025-12-04T14:26:42.3174130Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3174175Z Traceback (most recent call last):
2025-12-04T14:26:42.3174335Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3174377Z     getattr(self, test_name)()
2025-12-04T14:26:42.3174536Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3174570Z     fn()
2025-12-04T14:26:42.3174720Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3174760Z     method(*args, **kwargs)
2025-12-04T14:26:42.3174908Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3174947Z     method(*args, **kwargs)
2025-12-04T14:26:42.3175096Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3175133Z     with policy():
2025-12-04T14:26:42.3175282Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3175323Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3175727Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 67072 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3175741Z 
2025-12-04T14:26:42.3175813Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3176100Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3176113Z 
2025-12-04T14:26:42.3176199Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3176262Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3176323Z ======================= 1 failed, 19 deselected in 7.68s =======================
2025-12-04T14:26:42.3176361Z Got exit code 1
2025-12-04T14:26:42.3176588Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda
2025-12-04T14:26:42.3176718Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.3176953Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-09efae78b12893bf.xml
2025-12-04T14:26:42.3177014Z ============================= test session starts ==============================
2025-12-04T14:26:42.3177126Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3177168Z cachedir: .pytest_cache
2025-12-04T14:26:42.3177324Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3177373Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3177412Z configfile: pytest.ini
2025-12-04T14:26:42.3177575Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3177650Z collecting ... collected 60 items / 15 deselected / 45 selected
2025-12-04T14:26:42.3177704Z stepcurrent: skipping 15 already run items.
2025-12-04T14:26:42.3177747Z Running 5 items in this shard
2025-12-04T14:26:42.3177750Z 
2025-12-04T14:26:42.3178104Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda I1204 14:23:41.258000 395954 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 396023
2025-12-04T14:26:42.3178259Z I1204 14:23:41.258000 395954 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 396024
2025-12-04T14:26:42.3178411Z I1204 14:23:41.259000 395954 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 396025
2025-12-04T14:26:42.3178562Z I1204 14:23:41.260000 395954 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 396026
2025-12-04T14:26:42.3179144Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3179183Z   _warn_cpu_init()
2025-12-04T14:26:42.3179493Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3179536Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3180108Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3180184Z   _warn_cpu_init()
2025-12-04T14:26:42.3180751Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3180787Z   _warn_cpu_init()
2025-12-04T14:26:42.3181352Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3181389Z   _warn_cpu_init()
2025-12-04T14:26:42.3181532Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3181697Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3181992Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3182149Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3182438Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3182564Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3182844Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3182992Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3183270Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3183417Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3183695Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3183844Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3184122Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3184281Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3184819Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3184949Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3185145Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3185550Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3185667Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3185877Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3186044Z [rank1]:E1204 14:23:47.093000 396024 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3186081Z dist init r=1, world=4
2025-12-04T14:26:42.3186219Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3186379Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3186667Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3186857Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3187145Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3187271Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3187548Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3187695Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3187971Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3188118Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3188405Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3188558Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3188853Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3189016Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3189539Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 23040 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3189654Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3189848Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3190251Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3190365Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3190577Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3190741Z [rank3]:E1204 14:23:47.115000 396026 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3190779Z dist init r=3, world=4
2025-12-04T14:26:42.3190917Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3191077Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3191364Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3191518Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3191806Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3191931Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3192211Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3192358Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3192649Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3192805Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3193094Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3193240Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3193520Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3193668Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3194190Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 27136 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3194306Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3194500Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3194902Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3195015Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3195225Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3195389Z [rank2]:E1204 14:23:47.155000 396025 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3195426Z dist init r=2, world=4
2025-12-04T14:26:42.3195564Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3195724Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3196013Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3196166Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3196455Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3196578Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3197003Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3197165Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3197455Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3197614Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3197890Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3198028Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3198306Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3198453Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3198976Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 25088 on device 0. CUDA driver allocated memory was 2453667840 and is now 3571449856.
2025-12-04T14:26:42.3199091Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3199286Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3199687Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3199801Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3200011Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3200175Z [rank0]:E1204 14:23:47.190000 396023 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3200213Z dist init r=0, world=4
2025-12-04T14:26:42.3200548Z [rank0]:[W1204 14:23:47.057394445 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3200588Z FAILED [7.4141s] [ 20%]
2025-12-04T14:26:42.3200591Z 
2025-12-04T14:26:42.3200648Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3200787Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda _
2025-12-04T14:26:42.3200833Z Traceback (most recent call last):
2025-12-04T14:26:42.3201006Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3201051Z     self._join_processes(fn)
2025-12-04T14:26:42.3201225Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3201295Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3201474Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3201517Z     raise RuntimeError(error)
2025-12-04T14:26:42.3201622Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3201670Z Traceback (most recent call last):
2025-12-04T14:26:42.3201830Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3201872Z     getattr(self, test_name)()
2025-12-04T14:26:42.3202030Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3202064Z     fn()
2025-12-04T14:26:42.3202216Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3202256Z     method(*args, **kwargs)
2025-12-04T14:26:42.3202406Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3202445Z     method(*args, **kwargs)
2025-12-04T14:26:42.3202597Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3202633Z     with policy():
2025-12-04T14:26:42.3202784Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3202824Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3203220Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3203223Z 
2025-12-04T14:26:42.3203297Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3203576Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3203579Z 
2025-12-04T14:26:42.3203666Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3203668Z 
2025-12-04T14:26:42.3203670Z 
2025-12-04T14:26:42.3203745Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3203834Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3204069Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-09efae78b12893bf.xml -
2025-12-04T14:26:42.3204130Z =========================== short test summary info ============================
2025-12-04T14:26:42.3204418Z FAILED [7.4141s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3204466Z Traceback (most recent call last):
2025-12-04T14:26:42.3204630Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3204672Z     getattr(self, test_name)()
2025-12-04T14:26:42.3204832Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3204886Z     fn()
2025-12-04T14:26:42.3205037Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3205094Z     method(*args, **kwargs)
2025-12-04T14:26:42.3205242Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3205282Z     method(*args, **kwargs)
2025-12-04T14:26:42.3205443Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3205493Z     with policy():
2025-12-04T14:26:42.3205646Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3205686Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3206084Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3206087Z 
2025-12-04T14:26:42.3206160Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3206434Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3206437Z 
2025-12-04T14:26:42.3206523Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3206586Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3206646Z ======================= 1 failed, 15 deselected in 7.58s =======================
2025-12-04T14:26:42.3206684Z Got exit code 1
2025-12-04T14:26:42.3206723Z Retrying single test...
2025-12-04T14:26:42.3206966Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-572184dcfd664d5b.xml
2025-12-04T14:26:42.3207025Z ============================= test session starts ==============================
2025-12-04T14:26:42.3207138Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3207180Z cachedir: .pytest_cache
2025-12-04T14:26:42.3207338Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3207385Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3207425Z configfile: pytest.ini
2025-12-04T14:26:42.3207588Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3207660Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3207930Z stepcurrent: skipping 15 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3207973Z Running 1 items in this shard
2025-12-04T14:26:42.3207975Z 
2025-12-04T14:26:42.3208322Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda I1204 14:23:51.231000 396356 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 396425
2025-12-04T14:26:42.3208477Z I1204 14:23:51.232000 396356 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 396426
2025-12-04T14:26:42.3208629Z I1204 14:23:51.233000 396356 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 396427
2025-12-04T14:26:42.3208798Z I1204 14:23:51.234000 396356 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 396428
2025-12-04T14:26:42.3209376Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3209443Z   _warn_cpu_init()
2025-12-04T14:26:42.3210021Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3210060Z   _warn_cpu_init()
2025-12-04T14:26:42.3210350Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3210394Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3210962Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3211000Z   _warn_cpu_init()
2025-12-04T14:26:42.3211567Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3211604Z   _warn_cpu_init()
2025-12-04T14:26:42.3211750Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3211912Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3212204Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3212361Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3212649Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3212776Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3213054Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3213207Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3213503Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3213664Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3213951Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3214103Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3214383Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3214530Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3215057Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3215174Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3215369Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3215772Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3215886Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3216100Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3216264Z [rank1]:E1204 14:23:57.137000 396426 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3216304Z dist init r=1, world=4
2025-12-04T14:26:42.3216442Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3216602Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3216932Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3217088Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3217379Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3217502Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3217793Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3217954Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3218249Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3218410Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3218685Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3218822Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3219100Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3219248Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3219770Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 16896 on device 0. CUDA driver allocated memory was 2453667840 and is now 3571449856.
2025-12-04T14:26:42.3219885Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3220082Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3220484Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3220599Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3220811Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3220977Z [rank0]:E1204 14:23:57.198000 396425 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3221016Z dist init r=0, world=4
2025-12-04T14:26:42.3221154Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3221312Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3221603Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3221757Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3222068Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3222212Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3222504Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3222670Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3222945Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3223093Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3223369Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3223503Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3223782Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3223929Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3224452Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 16896 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3224567Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3224764Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3225164Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3225276Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3225489Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3225651Z [rank3]:E1204 14:23:57.199000 396428 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3225693Z dist init r=3, world=4
2025-12-04T14:26:42.3225830Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3225989Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3226292Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3226463Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3226822Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3226969Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3227246Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3227394Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3227671Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3227818Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3228096Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3228232Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3228510Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3228659Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3229183Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 18944 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3229298Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3229494Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3229894Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3230009Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3230221Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3230385Z [rank2]:E1204 14:23:57.207000 396427 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3230422Z dist init r=2, world=4
2025-12-04T14:26:42.3230772Z [rank0]:[W1204 14:23:57.967706725 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3230824Z FAILED [7.5129s] [100%]
2025-12-04T14:26:42.3230828Z 
2025-12-04T14:26:42.3230883Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3231033Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda _
2025-12-04T14:26:42.3231092Z Traceback (most recent call last):
2025-12-04T14:26:42.3231255Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3231297Z     self._join_processes(fn)
2025-12-04T14:26:42.3231472Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3231525Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3231703Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3231746Z     raise RuntimeError(error)
2025-12-04T14:26:42.3231826Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3231870Z Traceback (most recent call last):
2025-12-04T14:26:42.3232031Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3232073Z     getattr(self, test_name)()
2025-12-04T14:26:42.3232231Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3232264Z     fn()
2025-12-04T14:26:42.3232417Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3232456Z     method(*args, **kwargs)
2025-12-04T14:26:42.3232606Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3232646Z     method(*args, **kwargs)
2025-12-04T14:26:42.3232796Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3232833Z     with policy():
2025-12-04T14:26:42.3232986Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3233026Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3233422Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3233425Z 
2025-12-04T14:26:42.3233499Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3233775Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3233778Z 
2025-12-04T14:26:42.3233867Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3233870Z 
2025-12-04T14:26:42.3233872Z 
2025-12-04T14:26:42.3233949Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3234036Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3234269Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-572184dcfd664d5b.xml -
2025-12-04T14:26:42.3234341Z =========================== short test summary info ============================
2025-12-04T14:26:42.3234629Z FAILED [7.5129s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3234687Z Traceback (most recent call last):
2025-12-04T14:26:42.3234851Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3234924Z     getattr(self, test_name)()
2025-12-04T14:26:42.3235085Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3235119Z     fn()
2025-12-04T14:26:42.3235270Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3235310Z     method(*args, **kwargs)
2025-12-04T14:26:42.3235460Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3235500Z     method(*args, **kwargs)
2025-12-04T14:26:42.3235650Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3235686Z     with policy():
2025-12-04T14:26:42.3235837Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3235879Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3236273Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 12800 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3236276Z 
2025-12-04T14:26:42.3236348Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3236622Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3236625Z 
2025-12-04T14:26:42.3236712Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3236822Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3236884Z ======================= 1 failed, 19 deselected in 7.68s =======================
2025-12-04T14:26:42.3236921Z Got exit code 1
2025-12-04T14:26:42.3236961Z Retrying single test...
2025-12-04T14:26:42.3237151Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-3de0aab34e7a860c.xml
2025-12-04T14:26:42.3237209Z ============================= test session starts ==============================
2025-12-04T14:26:42.3237320Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3237362Z cachedir: .pytest_cache
2025-12-04T14:26:42.3237520Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3237566Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3237605Z configfile: pytest.ini
2025-12-04T14:26:42.3237769Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3237841Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3238110Z stepcurrent: skipping 15 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3238168Z Running 1 items in this shard
2025-12-04T14:26:42.3238170Z 
2025-12-04T14:26:42.3238521Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda I1204 14:24:01.217000 396758 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 396827
2025-12-04T14:26:42.3238691Z I1204 14:24:01.218000 396758 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 396828
2025-12-04T14:26:42.3238872Z I1204 14:24:01.218000 396758 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 396829
2025-12-04T14:26:42.3239023Z I1204 14:24:01.219000 396758 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 396830
2025-12-04T14:26:42.3239601Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3239640Z   _warn_cpu_init()
2025-12-04T14:26:42.3240211Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3240249Z   _warn_cpu_init()
2025-12-04T14:26:42.3240814Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3240851Z   _warn_cpu_init()
2025-12-04T14:26:42.3241415Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3241452Z   _warn_cpu_init()
2025-12-04T14:26:42.3241742Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3241786Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3241929Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3242093Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3242384Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3242539Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3242834Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3242981Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3243268Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3243428Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3243705Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3243852Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3244129Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3244266Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3244548Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3244697Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3245218Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 18944 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3245335Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3245531Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3245935Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3246048Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3246261Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3246425Z [rank2]:E1204 14:24:07.093000 396829 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3246464Z dist init r=2, world=4
2025-12-04T14:26:42.3246602Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3246806Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3247107Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3247276Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3247574Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3247712Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3247988Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3248137Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3248413Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3248559Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3248835Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3248971Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3249251Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3249400Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3249922Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 20992 on device 0. CUDA driver allocated memory was 2453667840 and is now 3571449856.
2025-12-04T14:26:42.3250037Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3250235Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3250637Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3250751Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3250963Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3251127Z [rank0]:E1204 14:24:07.106000 396827 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3251176Z dist init r=0, world=4
2025-12-04T14:26:42.3251313Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3251484Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3251790Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3251955Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3252238Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3252362Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3252640Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3252787Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3253065Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3253210Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3253486Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3253622Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3253902Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3254050Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3254571Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 14848 on device 3. CUDA driver allocated memory was 2250244096 and is now 3368026112.
2025-12-04T14:26:42.3254685Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3254879Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3255281Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3255392Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3255616Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3255790Z [rank3]:E1204 14:24:07.138000 396830 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3255828Z dist init r=3, world=4
2025-12-04T14:26:42.3255966Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3256147Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3256434Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3256587Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3256900Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3257024Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3257303Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3257451Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3257731Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3257880Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3258156Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3258294Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3258571Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3258720Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3259236Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 20992 on device 1. CUDA driver allocated memory was 2317352960 and is now 3435134976.
2025-12-04T14:26:42.3259354Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3259550Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3259966Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3260094Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3260307Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3260501Z [rank1]:E1204 14:24:07.170000 396828 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3260539Z dist init r=1, world=4
2025-12-04T14:26:42.3260873Z [rank0]:[W1204 14:24:07.801825411 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3260912Z FAILED [7.5140s] [100%]
2025-12-04T14:26:42.3260916Z 
2025-12-04T14:26:42.3260971Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3261112Z _ TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda _
2025-12-04T14:26:42.3261157Z Traceback (most recent call last):
2025-12-04T14:26:42.3261320Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3261365Z     self._join_processes(fn)
2025-12-04T14:26:42.3261538Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3261592Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3261770Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3261812Z     raise RuntimeError(error)
2025-12-04T14:26:42.3261892Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3261937Z Traceback (most recent call last):
2025-12-04T14:26:42.3262098Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3262139Z     getattr(self, test_name)()
2025-12-04T14:26:42.3262299Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3262334Z     fn()
2025-12-04T14:26:42.3262489Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3262529Z     method(*args, **kwargs)
2025-12-04T14:26:42.3262681Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3262721Z     method(*args, **kwargs)
2025-12-04T14:26:42.3262870Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3262908Z     with policy():
2025-12-04T14:26:42.3263058Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3263099Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3263494Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 20992 on device 0. CUDA driver allocated memory was 2453667840 and is now 3571449856.
2025-12-04T14:26:42.3263497Z 
2025-12-04T14:26:42.3263572Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3263857Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3263870Z 
2025-12-04T14:26:42.3263959Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3263961Z 
2025-12-04T14:26:42.3264020Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.3264065Z Traceback (most recent call last):
2025-12-04T14:26:42.3264250Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3264292Z     getattr(self, test_name)()
2025-12-04T14:26:42.3264449Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3264482Z     fn()
2025-12-04T14:26:42.3264633Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3264672Z     method(*args, **kwargs)
2025-12-04T14:26:42.3264823Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3264862Z     method(*args, **kwargs)
2025-12-04T14:26:42.3265011Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3265047Z     with policy():
2025-12-04T14:26:42.3265199Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3265239Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3265630Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 18944 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3265632Z 
2025-12-04T14:26:42.3265704Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3265979Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3265981Z 
2025-12-04T14:26:42.3266068Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3266071Z 
2025-12-04T14:26:42.3266073Z 
2025-12-04T14:26:42.3266148Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3266236Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3266471Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-3de0aab34e7a860c.xml -
2025-12-04T14:26:42.3266532Z =========================== short test summary info ============================
2025-12-04T14:26:42.3266858Z FAILED [7.5140s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3266906Z Traceback (most recent call last):
2025-12-04T14:26:42.3267068Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3267112Z     getattr(self, test_name)()
2025-12-04T14:26:42.3267272Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3267306Z     fn()
2025-12-04T14:26:42.3267456Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3267510Z     method(*args, **kwargs)
2025-12-04T14:26:42.3267661Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3267720Z     method(*args, **kwargs)
2025-12-04T14:26:42.3267869Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3267905Z     with policy():
2025-12-04T14:26:42.3268069Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3268122Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3268516Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 20992 on device 0. CUDA driver allocated memory was 2453667840 and is now 3571449856.
2025-12-04T14:26:42.3268518Z 
2025-12-04T14:26:42.3268591Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3268862Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3268866Z 
2025-12-04T14:26:42.3268951Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3268954Z 
2025-12-04T14:26:42.3269014Z Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.3269060Z Traceback (most recent call last):
2025-12-04T14:26:42.3269221Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3269263Z     getattr(self, test_name)()
2025-12-04T14:26:42.3269421Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3269456Z     fn()
2025-12-04T14:26:42.3269605Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3269645Z     method(*args, **kwargs)
2025-12-04T14:26:42.3269793Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3269833Z     method(*args, **kwargs)
2025-12-04T14:26:42.3269982Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3270020Z     with policy():
2025-12-04T14:26:42.3270169Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3270209Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3270598Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda! Caching allocator allocated memory was 512 and is now reported as 18944 on device 2. CUDA driver allocated memory was 2300575744 and is now 3418357760.
2025-12-04T14:26:42.3270602Z 
2025-12-04T14:26:42.3270674Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3270945Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3270951Z 
2025-12-04T14:26:42.3271037Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3271099Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3271160Z ======================= 1 failed, 19 deselected in 7.67s =======================
2025-12-04T14:26:42.3271198Z Got exit code 1
2025-12-04T14:26:42.3271432Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda
2025-12-04T14:26:42.3271574Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.3271764Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-110ef887ac58b403.xml
2025-12-04T14:26:42.3271822Z ============================= test session starts ==============================
2025-12-04T14:26:42.3271955Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3271997Z cachedir: .pytest_cache
2025-12-04T14:26:42.3272153Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3272199Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3272239Z configfile: pytest.ini
2025-12-04T14:26:42.3272401Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3272474Z collecting ... collected 60 items / 16 deselected / 44 selected
2025-12-04T14:26:42.3272528Z stepcurrent: skipping 16 already run items.
2025-12-04T14:26:42.3272571Z Running 4 items in this shard
2025-12-04T14:26:42.3272573Z 
2025-12-04T14:26:42.3272878Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda I1204 14:24:11.226000 397160 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 397229
2025-12-04T14:26:42.3273033Z I1204 14:24:11.227000 397160 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 397230
2025-12-04T14:26:42.3273183Z I1204 14:24:11.228000 397160 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 397231
2025-12-04T14:26:42.3273334Z I1204 14:24:11.228000 397160 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 397232
2025-12-04T14:26:42.3273696Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3273749Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3274106Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3274154Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3274504Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3274551Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3274902Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3274947Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3275228Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3275271Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3275858Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3275907Z   _warn_cpu_init()
2025-12-04T14:26:42.3276205Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3276261Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3276537Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3276580Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3277188Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3277228Z   _warn_cpu_init()
2025-12-04T14:26:42.3277503Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3277545Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3278114Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3278151Z   _warn_cpu_init()
2025-12-04T14:26:42.3278426Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3278469Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3279040Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3279077Z   _warn_cpu_init()
2025-12-04T14:26:42.3279362Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3279400Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3279685Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3279724Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3280022Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3280060Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3280290Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3280347Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3280582Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3280639Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3280859Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3280900Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3281122Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3281164Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3281386Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3281427Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3281647Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3281687Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3281905Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3281944Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3282163Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3282203Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3282495Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3282536Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3283811Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3283941Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3285235Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3285379Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3286644Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3286803Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3288068Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3288191Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3288335Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3288499Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3288793Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3288952Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3289253Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3289379Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3289673Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3289836Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3290127Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3290275Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3290549Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3290687Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3290965Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3291114Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3291596Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1776128 on device 1. CUDA driver allocated memory was 2317352960 and is now 3944742912.
2025-12-04T14:26:42.3291713Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3291909Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3292266Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3292379Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3292592Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3292755Z [rank1]:E1204 14:24:19.955000 397230 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3292894Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3293053Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3293340Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3293506Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3293794Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3293932Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3294220Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3294378Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3294654Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3294801Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3295076Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3295214Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3295492Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3295640Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3296118Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1776128 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3296233Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3296429Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3296817Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3296931Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3297144Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3297308Z [rank0]:E1204 14:24:19.955000 397229 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3297350Z dist init r=1, world=4
2025-12-04T14:26:42.3297388Z dist init r=0, world=4
2025-12-04T14:26:42.3297526Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3297684Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3297986Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3298153Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3298455Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3298602Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3298878Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3299025Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3299301Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3299447Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3299722Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3299859Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3300137Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3300284Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3300762Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1776128 on device 3. CUDA driver allocated memory was 2250244096 and is now 3877634048.
2025-12-04T14:26:42.3300876Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3301072Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3301422Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3301535Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3301748Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3301911Z [rank3]:E1204 14:24:19.993000 397232 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3301949Z dist init r=3, world=4
2025-12-04T14:26:42.3302100Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3302259Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3302558Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3302735Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3303019Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3303144Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3303420Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3303567Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3303846Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3303992Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3304269Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3304404Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3304680Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3304831Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3305308Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1776128 on device 2. CUDA driver allocated memory was 2300575744 and is now 3927965696.
2025-12-04T14:26:42.3305423Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3305619Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3305971Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3306084Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3306310Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3306474Z [rank2]:E1204 14:24:20.003000 397231 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3306523Z dist init r=2, world=4
2025-12-04T14:26:42.3306901Z [rank0]:[W1204 14:24:20.661993074 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3306973Z FAILED [10.6168s] [ 25%]
2025-12-04T14:26:42.3306975Z 
2025-12-04T14:26:42.3307034Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3307133Z ______ TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda ______
2025-12-04T14:26:42.3307179Z Traceback (most recent call last):
2025-12-04T14:26:42.3307343Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3307387Z     self._join_processes(fn)
2025-12-04T14:26:42.3307560Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3307615Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3307792Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3307838Z     raise RuntimeError(error)
2025-12-04T14:26:42.3307918Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3307963Z Traceback (most recent call last):
2025-12-04T14:26:42.3308123Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3308166Z     getattr(self, test_name)()
2025-12-04T14:26:42.3308325Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3308359Z     fn()
2025-12-04T14:26:42.3308512Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3308552Z     method(*args, **kwargs)
2025-12-04T14:26:42.3308702Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3308742Z     method(*args, **kwargs)
2025-12-04T14:26:42.3308893Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3308931Z     with policy():
2025-12-04T14:26:42.3309083Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3309122Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3309472Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1776128 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3309475Z 
2025-12-04T14:26:42.3309550Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3309776Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3309780Z 
2025-12-04T14:26:42.3309867Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3309870Z 
2025-12-04T14:26:42.3309872Z 
2025-12-04T14:26:42.3309945Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3310033Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3310282Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-110ef887ac58b403.xml -
2025-12-04T14:26:42.3310357Z =========================== short test summary info ============================
2025-12-04T14:26:42.3310600Z FAILED [10.6168s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3310672Z Traceback (most recent call last):
2025-12-04T14:26:42.3310835Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3310878Z     getattr(self, test_name)()
2025-12-04T14:26:42.3311036Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3311070Z     fn()
2025-12-04T14:26:42.3311221Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3311261Z     method(*args, **kwargs)
2025-12-04T14:26:42.3311411Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3311451Z     method(*args, **kwargs)
2025-12-04T14:26:42.3311599Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3311638Z     with policy():
2025-12-04T14:26:42.3311789Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3311830Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3312181Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1776128 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3312184Z 
2025-12-04T14:26:42.3312258Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3312483Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3312485Z 
2025-12-04T14:26:42.3312571Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3312634Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3312696Z ====================== 1 failed, 16 deselected in 10.78s =======================
2025-12-04T14:26:42.3312733Z Got exit code 1
2025-12-04T14:26:42.3312773Z Retrying single test...
2025-12-04T14:26:42.3312962Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-de818e7b10407058.xml
2025-12-04T14:26:42.3313019Z ============================= test session starts ==============================
2025-12-04T14:26:42.3313132Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3313172Z cachedir: .pytest_cache
2025-12-04T14:26:42.3313329Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3313373Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3313416Z configfile: pytest.ini
2025-12-04T14:26:42.3313578Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3313652Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3313882Z stepcurrent: skipping 16 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3313926Z Running 1 items in this shard
2025-12-04T14:26:42.3313928Z 
2025-12-04T14:26:42.3314231Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda I1204 14:24:24.226000 397562 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 397631
2025-12-04T14:26:42.3314398Z I1204 14:24:24.227000 397562 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 397632
2025-12-04T14:26:42.3314580Z I1204 14:24:24.227000 397562 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 397633
2025-12-04T14:26:42.3314732Z I1204 14:24:24.228000 397562 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 397634
2025-12-04T14:26:42.3315092Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3315142Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3315495Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3315543Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3315893Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3315940Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3316288Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3316335Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3316615Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3316662Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3317287Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3317325Z   _warn_cpu_init()
2025-12-04T14:26:42.3317614Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3317653Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3317933Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3317976Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3318563Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3318616Z   _warn_cpu_init()
2025-12-04T14:26:42.3318901Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3318954Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3319246Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3319290Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3319563Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3319606Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3320178Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3320217Z   _warn_cpu_init()
2025-12-04T14:26:42.3320784Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3320821Z   _warn_cpu_init()
2025-12-04T14:26:42.3321108Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3321146Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3321433Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3321471Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3321702Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3321744Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3321970Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3322012Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3322234Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3322277Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3322497Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3322538Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3322767Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3322808Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3323039Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3323079Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3323307Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3323359Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3323578Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3323618Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3323910Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3323954Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3325227Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3325353Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3326618Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3326790Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3328068Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3328227Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3329494Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3329614Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3329758Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3329923Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3330215Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3330372Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3330662Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3330786Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3331069Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3331220Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3331498Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3331647Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3331923Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3332074Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3332352Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3332517Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3333014Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3333131Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3333327Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3333682Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3333797Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3334007Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3334172Z [rank0]:E1204 14:24:33.151000 397631 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3334210Z dist init r=0, world=4
2025-12-04T14:26:42.3334349Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3334509Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3334797Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3334953Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3335236Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3335360Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3335636Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3335785Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3336061Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3336219Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3336494Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3336642Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3336982Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3337142Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3337615Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 1. CUDA driver allocated memory was 2317352960 and is now 3944742912.
2025-12-04T14:26:42.3337730Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3337925Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3338282Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3338394Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3338607Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3338771Z [rank1]:E1204 14:24:33.156000 397632 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3338809Z dist init r=1, world=4
2025-12-04T14:26:42.3338945Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3339107Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3339392Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3339546Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3339831Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3339955Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3340233Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3340380Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3340673Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3340831Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3341106Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3341266Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3341542Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3341691Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3342167Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 2. CUDA driver allocated memory was 2300575744 and is now 3927965696.
2025-12-04T14:26:42.3342285Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3342480Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3342832Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3342946Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3343157Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3343324Z [rank2]:E1204 14:24:33.175000 397633 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3343362Z dist init r=2, world=4
2025-12-04T14:26:42.3343501Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3343659Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3343947Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3344104Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3344392Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3344517Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3344804Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3344952Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3345242Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3345401Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3345688Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3345826Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3346103Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3346252Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3346729Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1776128 on device 3. CUDA driver allocated memory was 2250244096 and is now 3877634048.
2025-12-04T14:26:42.3346876Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3347072Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3347423Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3347536Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3347748Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3347911Z [rank3]:E1204 14:24:33.229000 397634 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3347951Z dist init r=3, world=4
2025-12-04T14:26:42.3348286Z [rank0]:[W1204 14:24:33.814537327 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3348328Z FAILED [10.9175s] [100%]
2025-12-04T14:26:42.3348329Z 
2025-12-04T14:26:42.3348385Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3348485Z ______ TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda ______
2025-12-04T14:26:42.3348532Z Traceback (most recent call last):
2025-12-04T14:26:42.3348695Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3348737Z     self._join_processes(fn)
2025-12-04T14:26:42.3348932Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3348987Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3349165Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3349223Z     raise RuntimeError(error)
2025-12-04T14:26:42.3349303Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3349347Z Traceback (most recent call last):
2025-12-04T14:26:42.3349524Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3349582Z     getattr(self, test_name)()
2025-12-04T14:26:42.3349739Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3349774Z     fn()
2025-12-04T14:26:42.3349925Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3349967Z     method(*args, **kwargs)
2025-12-04T14:26:42.3350116Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3350157Z     method(*args, **kwargs)
2025-12-04T14:26:42.3350306Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3350344Z     with policy():
2025-12-04T14:26:42.3350496Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3350537Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3350888Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3350891Z 
2025-12-04T14:26:42.3350967Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3351196Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3351199Z 
2025-12-04T14:26:42.3351288Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3351290Z 
2025-12-04T14:26:42.3351351Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3351396Z Traceback (most recent call last):
2025-12-04T14:26:42.3351559Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3351600Z     getattr(self, test_name)()
2025-12-04T14:26:42.3351758Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3351793Z     fn()
2025-12-04T14:26:42.3351942Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3351982Z     method(*args, **kwargs)
2025-12-04T14:26:42.3352132Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3352170Z     method(*args, **kwargs)
2025-12-04T14:26:42.3352319Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3352356Z     with policy():
2025-12-04T14:26:42.3352507Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3352547Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3352907Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 1. CUDA driver allocated memory was 2317352960 and is now 3944742912.
2025-12-04T14:26:42.3352919Z 
2025-12-04T14:26:42.3352993Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3353215Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3353217Z 
2025-12-04T14:26:42.3353328Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3353331Z 
2025-12-04T14:26:42.3353332Z 
2025-12-04T14:26:42.3353410Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3353498Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3353731Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-de818e7b10407058.xml -
2025-12-04T14:26:42.3353791Z =========================== short test summary info ============================
2025-12-04T14:26:42.3354034Z FAILED [10.9175s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3354080Z Traceback (most recent call last):
2025-12-04T14:26:42.3354243Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3354285Z     getattr(self, test_name)()
2025-12-04T14:26:42.3354444Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3354477Z     fn()
2025-12-04T14:26:42.3354627Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3354666Z     method(*args, **kwargs)
2025-12-04T14:26:42.3354817Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3354857Z     method(*args, **kwargs)
2025-12-04T14:26:42.3355006Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3355041Z     with policy():
2025-12-04T14:26:42.3355195Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3355236Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3355584Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3355586Z 
2025-12-04T14:26:42.3355658Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3355883Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3355886Z 
2025-12-04T14:26:42.3355972Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3355974Z 
2025-12-04T14:26:42.3356033Z Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3356080Z Traceback (most recent call last):
2025-12-04T14:26:42.3356240Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3356282Z     getattr(self, test_name)()
2025-12-04T14:26:42.3356438Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3356472Z     fn()
2025-12-04T14:26:42.3356632Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3356685Z     method(*args, **kwargs)
2025-12-04T14:26:42.3356879Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3356918Z     method(*args, **kwargs)
2025-12-04T14:26:42.3357067Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3357135Z     with policy():
2025-12-04T14:26:42.3357286Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3357327Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3357673Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 1. CUDA driver allocated memory was 2317352960 and is now 3944742912.
2025-12-04T14:26:42.3357676Z 
2025-12-04T14:26:42.3357749Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3357971Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3357973Z 
2025-12-04T14:26:42.3358060Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3358124Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3358187Z ====================== 1 failed, 19 deselected in 11.09s =======================
2025-12-04T14:26:42.3358224Z Got exit code 1
2025-12-04T14:26:42.3358264Z Retrying single test...
2025-12-04T14:26:42.3358455Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d1668f4dedfa2621.xml
2025-12-04T14:26:42.3358513Z ============================= test session starts ==============================
2025-12-04T14:26:42.3358626Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3358666Z cachedir: .pytest_cache
2025-12-04T14:26:42.3358824Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3358870Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3358911Z configfile: pytest.ini
2025-12-04T14:26:42.3359073Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3359147Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3359365Z stepcurrent: skipping 16 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3359409Z Running 1 items in this shard
2025-12-04T14:26:42.3359411Z 
2025-12-04T14:26:42.3359714Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda I1204 14:24:37.661000 397964 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 398033
2025-12-04T14:26:42.3359869Z I1204 14:24:37.662000 397964 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 398034
2025-12-04T14:26:42.3360023Z I1204 14:24:37.662000 397964 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 398035
2025-12-04T14:26:42.3360174Z I1204 14:24:37.663000 397964 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 398036
2025-12-04T14:26:42.3360552Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3360614Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3360966Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3361044Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3361396Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3361443Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3361794Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3361841Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3362123Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3362170Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3362750Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3362789Z   _warn_cpu_init()
2025-12-04T14:26:42.3363080Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3363120Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3363398Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3363441Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3363714Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3363757Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3364329Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3364369Z   _warn_cpu_init()
2025-12-04T14:26:42.3364949Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3364987Z   _warn_cpu_init()
2025-12-04T14:26:42.3365276Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3365319Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3365901Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3365951Z   _warn_cpu_init()
2025-12-04T14:26:42.3366238Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3366279Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3366563Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3366601Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3366926Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3366965Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3367195Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3367236Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3367460Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3367502Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3367728Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3367769Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3367991Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3368031Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3368252Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3368291Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3368511Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3368551Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3368771Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3368811Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3369029Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3369070Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3369373Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3369428Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3370715Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3370856Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3372119Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3372243Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3373508Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3373631Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3374911Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.)
2025-12-04T14:26:42.3375056Z   return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-12-04T14:26:42.3375199Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3375364Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3375657Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3375816Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3376105Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3376230Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3376511Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3376661Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3376980Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3377128Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3377403Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3377544Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3377824Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3377973Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3378451Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 1. CUDA driver allocated memory was 2317352960 and is now 3944742912.
2025-12-04T14:26:42.3378580Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3380588Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3380970Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3381126Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3381342Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3381511Z [rank1]:E1204 14:24:46.415000 398034 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3381552Z dist init r=1, world=4
2025-12-04T14:26:42.3381692Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3381855Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3382145Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3382302Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3382588Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3382713Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3382995Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3383144Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3383422Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3383569Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3383847Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3383990Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3384276Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3384426Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3384914Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3385042Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3385237Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3385614Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3385727Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3385940Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3386107Z [rank0]:E1204 14:24:46.417000 398033 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3386145Z dist init r=0, world=4
2025-12-04T14:26:42.3386283Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3386446Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3386735Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3386928Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3387213Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3387336Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3387615Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3387763Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3388040Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3388188Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3388465Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3388603Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3388884Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3389055Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3389540Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1742336 on device 2. CUDA driver allocated memory was 2300575744 and is now 3927965696.
2025-12-04T14:26:42.3389682Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3389879Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3390230Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3390344Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3390554Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3390720Z [rank2]:E1204 14:24:46.420000 398035 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3390758Z dist init r=2, world=4
2025-12-04T14:26:42.3390895Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3391056Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3391343Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3391498Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3391781Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3391906Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3392181Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3392328Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3392605Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3392753Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3393028Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3393176Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3393457Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3393617Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3394102Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1776128 on device 3. CUDA driver allocated memory was 2250244096 and is now 3877634048.
2025-12-04T14:26:42.3394229Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3394425Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3394777Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3394891Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3395104Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3395267Z [rank3]:E1204 14:24:46.423000 398036 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3395307Z dist init r=3, world=4
2025-12-04T14:26:42.3395644Z [rank0]:[W1204 14:24:46.101893468 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3395686Z FAILED [10.6179s] [100%]
2025-12-04T14:26:42.3395688Z 
2025-12-04T14:26:42.3395746Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3395847Z ______ TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda ______
2025-12-04T14:26:42.3395895Z Traceback (most recent call last):
2025-12-04T14:26:42.3396058Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3396102Z     self._join_processes(fn)
2025-12-04T14:26:42.3396275Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3396330Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3396511Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3396556Z     raise RuntimeError(error)
2025-12-04T14:26:42.3396636Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3396681Z Traceback (most recent call last):
2025-12-04T14:26:42.3396888Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3396931Z     getattr(self, test_name)()
2025-12-04T14:26:42.3397089Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3397124Z     fn()
2025-12-04T14:26:42.3397295Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3397337Z     method(*args, **kwargs)
2025-12-04T14:26:42.3397488Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3397548Z     method(*args, **kwargs)
2025-12-04T14:26:42.3397698Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3397735Z     with policy():
2025-12-04T14:26:42.3397916Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3397957Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3398311Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3398313Z 
2025-12-04T14:26:42.3398390Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3398617Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3398619Z 
2025-12-04T14:26:42.3398707Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3398710Z 
2025-12-04T14:26:42.3398712Z 
2025-12-04T14:26:42.3398789Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3398877Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3399112Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d1668f4dedfa2621.xml -
2025-12-04T14:26:42.3399173Z =========================== short test summary info ============================
2025-12-04T14:26:42.3399418Z FAILED [10.6179s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3399465Z Traceback (most recent call last):
2025-12-04T14:26:42.3399628Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3399671Z     getattr(self, test_name)()
2025-12-04T14:26:42.3399832Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3399867Z     fn()
2025-12-04T14:26:42.3400017Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3400056Z     method(*args, **kwargs)
2025-12-04T14:26:42.3400209Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3400249Z     method(*args, **kwargs)
2025-12-04T14:26:42.3400402Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3400440Z     with policy():
2025-12-04T14:26:42.3400591Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3400633Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3400985Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1708544 on device 0. CUDA driver allocated memory was 2453667840 and is now 4081057792.
2025-12-04T14:26:42.3400988Z 
2025-12-04T14:26:42.3401062Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3401296Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3401311Z 
2025-12-04T14:26:42.3401398Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3401460Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3401522Z ====================== 1 failed, 19 deselected in 10.77s =======================
2025-12-04T14:26:42.3401571Z Got exit code 1
2025-12-04T14:26:42.3401757Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda
2025-12-04T14:26:42.3401885Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.3402074Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d4786a53251b459e.xml
2025-12-04T14:26:42.3402133Z ============================= test session starts ==============================
2025-12-04T14:26:42.3402245Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3402288Z cachedir: .pytest_cache
2025-12-04T14:26:42.3402445Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3402491Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3402532Z configfile: pytest.ini
2025-12-04T14:26:42.3402698Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3402773Z collecting ... collected 60 items / 17 deselected / 43 selected
2025-12-04T14:26:42.3402825Z stepcurrent: skipping 17 already run items.
2025-12-04T14:26:42.3402869Z Running 3 items in this shard
2025-12-04T14:26:42.3402871Z 
2025-12-04T14:26:42.3403171Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda I1204 14:24:50.696000 398366 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 398435
2025-12-04T14:26:42.3403327Z I1204 14:24:50.697000 398366 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 398436
2025-12-04T14:26:42.3403477Z I1204 14:24:50.697000 398366 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 398437
2025-12-04T14:26:42.3403628Z I1204 14:24:50.698000 398366 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 398438
2025-12-04T14:26:42.3403988Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3404039Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3404392Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3404441Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3404793Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3404838Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3405204Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3405249Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3405851Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3405901Z   _warn_cpu_init()
2025-12-04T14:26:42.3406470Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3406509Z   _warn_cpu_init()
2025-12-04T14:26:42.3407124Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3407163Z   _warn_cpu_init()
2025-12-04T14:26:42.3407732Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3407770Z   _warn_cpu_init()
2025-12-04T14:26:42.3408060Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3408104Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3408251Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3408413Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3408705Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3408863Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3409151Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3409278Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3409557Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3409729Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3410020Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3410169Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3410471Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3410610Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3410887Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3411036Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3411507Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3411624Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3411821Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3412166Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3412281Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3412497Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3412661Z [rank3]:E1204 14:24:59.774000 398438 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3412700Z dist init r=3, world=4
2025-12-04T14:26:42.3412838Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3412997Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3413285Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3413440Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3413725Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3413868Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3414146Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3414309Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3414599Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3414757Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3415035Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3415170Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3415448Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3415597Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3416067Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 0. CUDA driver allocated memory was 2453667840 and is now 4055891968.
2025-12-04T14:26:42.3416182Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3416379Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3416727Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3416881Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3417097Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3417262Z [rank0]:E1204 14:24:59.788000 398435 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3417302Z dist init r=0, world=4
2025-12-04T14:26:42.3417439Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3417598Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3417888Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3418040Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3418348Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3418489Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3418765Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3418939Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3419216Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3419365Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3419643Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3419783Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3420062Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3420211Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3420681Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 295424 on device 2. CUDA driver allocated memory was 2300575744 and is now 3902799872.
2025-12-04T14:26:42.3420795Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3420991Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3421336Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3421451Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3421664Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3421831Z [rank2]:E1204 14:24:59.794000 398437 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3421868Z dist init r=2, world=4
2025-12-04T14:26:42.3422008Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3422167Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3422464Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3422618Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3422913Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3423047Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3423333Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3423481Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3423759Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3423906Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3424185Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3424321Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3424600Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3424746Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3425214Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 1. CUDA driver allocated memory was 2317352960 and is now 3919577088.
2025-12-04T14:26:42.3425329Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3425524Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3425870Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3425982Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3426197Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3426364Z [rank1]:E1204 14:24:59.822000 398436 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3426402Z dist init r=1, world=4
2025-12-04T14:26:42.3426787Z [rank0]:[W1204 14:25:00.555654687 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3426828Z FAILED [11.0162s] [ 33%]
2025-12-04T14:26:42.3426830Z 
2025-12-04T14:26:42.3426906Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3427004Z ________ TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda ________
2025-12-04T14:26:42.3427049Z Traceback (most recent call last):
2025-12-04T14:26:42.3427226Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3427296Z     self._join_processes(fn)
2025-12-04T14:26:42.3427468Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3427522Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3427701Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3427744Z     raise RuntimeError(error)
2025-12-04T14:26:42.3427824Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3427870Z Traceback (most recent call last):
2025-12-04T14:26:42.3428029Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3428072Z     getattr(self, test_name)()
2025-12-04T14:26:42.3428229Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3428266Z     fn()
2025-12-04T14:26:42.3428417Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3428457Z     method(*args, **kwargs)
2025-12-04T14:26:42.3428608Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3428648Z     method(*args, **kwargs)
2025-12-04T14:26:42.3428797Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3428835Z     with policy():
2025-12-04T14:26:42.3428985Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3429026Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3429367Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3429370Z 
2025-12-04T14:26:42.3429445Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3429664Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3429666Z 
2025-12-04T14:26:42.3429753Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3429756Z 
2025-12-04T14:26:42.3429757Z 
2025-12-04T14:26:42.3429832Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3429919Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3430157Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-d4786a53251b459e.xml -
2025-12-04T14:26:42.3430217Z =========================== short test summary info ============================
2025-12-04T14:26:42.3430455Z FAILED [11.0162s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3430513Z Traceback (most recent call last):
2025-12-04T14:26:42.3430677Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3430730Z     getattr(self, test_name)()
2025-12-04T14:26:42.3430889Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3430924Z     fn()
2025-12-04T14:26:42.3431087Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3431137Z     method(*args, **kwargs)
2025-12-04T14:26:42.3431287Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3431326Z     method(*args, **kwargs)
2025-12-04T14:26:42.3431476Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3431513Z     with policy():
2025-12-04T14:26:42.3431663Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3431705Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3432048Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3432051Z 
2025-12-04T14:26:42.3432125Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3432342Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3432344Z 
2025-12-04T14:26:42.3432430Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3432492Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3432555Z ====================== 1 failed, 17 deselected in 11.18s =======================
2025-12-04T14:26:42.3432592Z Got exit code 1
2025-12-04T14:26:42.3432631Z Retrying single test...
2025-12-04T14:26:42.3432818Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-46578d38ff802f9b.xml
2025-12-04T14:26:42.3432876Z ============================= test session starts ==============================
2025-12-04T14:26:42.3432989Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3433029Z cachedir: .pytest_cache
2025-12-04T14:26:42.3433187Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3433233Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3433273Z configfile: pytest.ini
2025-12-04T14:26:42.3433434Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3433510Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3433723Z stepcurrent: skipping 17 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3433768Z Running 1 items in this shard
2025-12-04T14:26:42.3433771Z 
2025-12-04T14:26:42.3434067Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda I1204 14:25:04.133000 398768 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 398837
2025-12-04T14:26:42.3434221Z I1204 14:25:04.134000 398768 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 398838
2025-12-04T14:26:42.3434385Z I1204 14:25:04.135000 398768 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 398839
2025-12-04T14:26:42.3434547Z I1204 14:25:04.135000 398768 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 398840
2025-12-04T14:26:42.3434914Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3434974Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3435327Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3435374Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3435728Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3435774Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3436125Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3436170Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3436790Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3436829Z   _warn_cpu_init()
2025-12-04T14:26:42.3437397Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3437436Z   _warn_cpu_init()
2025-12-04T14:26:42.3438000Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3438037Z   _warn_cpu_init()
2025-12-04T14:26:42.3438600Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3438639Z   _warn_cpu_init()
2025-12-04T14:26:42.3438944Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3439000Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3439143Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3439304Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3439621Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3439776Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3440064Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3440188Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3440469Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3440619Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3440894Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3441042Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3441318Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3441455Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3441733Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3441882Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3442352Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 1. CUDA driver allocated memory was 2317352960 and is now 3919577088.
2025-12-04T14:26:42.3442467Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3442665Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3443012Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3443140Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3443352Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3443529Z [rank1]:E1204 14:25:13.119000 398838 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3443568Z dist init r=1, world=4
2025-12-04T14:26:42.3443732Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3443892Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3444178Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3444332Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3444617Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3444743Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3445021Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3445170Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3445448Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3445595Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3445871Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3446007Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3446284Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3446431Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3446951Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3447067Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3447262Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3447647Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3447780Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3448011Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3448196Z [rank3]:E1204 14:25:13.120000 398840 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3448235Z dist init r=3, world=4
2025-12-04T14:26:42.3448371Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3448531Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3448817Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3448972Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3449258Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3449381Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3449660Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3449811Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3450090Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3450237Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3450511Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3450647Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3450924Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3451071Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3451541Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 2. CUDA driver allocated memory was 2300575744 and is now 3902799872.
2025-12-04T14:26:42.3451667Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3451863Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3452221Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3452359Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3452570Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3452736Z [rank2]:E1204 14:25:13.177000 398839 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3452773Z dist init r=2, world=4
2025-12-04T14:26:42.3452911Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3453070Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3453357Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3453511Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3453798Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3453921Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3454198Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3454347Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3454623Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3454770Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3455045Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3455182Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3455460Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3455608Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3456085Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 0. CUDA driver allocated memory was 2453667840 and is now 4055891968.
2025-12-04T14:26:42.3456208Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3456403Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3456814Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3456926Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3457139Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3457303Z [rank0]:E1204 14:25:13.181000 398837 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3457342Z dist init r=0, world=4
2025-12-04T14:26:42.3457675Z [rank0]:[W1204 14:25:13.976533448 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3457717Z FAILED [10.8191s] [100%]
2025-12-04T14:26:42.3457720Z 
2025-12-04T14:26:42.3457774Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3457872Z ________ TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda ________
2025-12-04T14:26:42.3457918Z Traceback (most recent call last):
2025-12-04T14:26:42.3458080Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3458124Z     self._join_processes(fn)
2025-12-04T14:26:42.3458297Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3458350Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3458529Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3458572Z     raise RuntimeError(error)
2025-12-04T14:26:42.3458652Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3458698Z Traceback (most recent call last):
2025-12-04T14:26:42.3458859Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3458902Z     getattr(self, test_name)()
2025-12-04T14:26:42.3459058Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3459094Z     fn()
2025-12-04T14:26:42.3459244Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3459285Z     method(*args, **kwargs)
2025-12-04T14:26:42.3459436Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3459476Z     method(*args, **kwargs)
2025-12-04T14:26:42.3459626Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3459664Z     with policy():
2025-12-04T14:26:42.3459833Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3459874Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3460214Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3460238Z 
2025-12-04T14:26:42.3460314Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3460561Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3460563Z 
2025-12-04T14:26:42.3460652Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3460654Z 
2025-12-04T14:26:42.3460656Z 
2025-12-04T14:26:42.3460732Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3460818Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3461052Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-46578d38ff802f9b.xml -
2025-12-04T14:26:42.3461113Z =========================== short test summary info ============================
2025-12-04T14:26:42.3461353Z FAILED [10.8191s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3461399Z Traceback (most recent call last):
2025-12-04T14:26:42.3461563Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3461605Z     getattr(self, test_name)()
2025-12-04T14:26:42.3461764Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3461800Z     fn()
2025-12-04T14:26:42.3461950Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3461991Z     method(*args, **kwargs)
2025-12-04T14:26:42.3462141Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3462180Z     method(*args, **kwargs)
2025-12-04T14:26:42.3462331Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3462369Z     with policy():
2025-12-04T14:26:42.3462518Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3462558Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3462900Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3462903Z 
2025-12-04T14:26:42.3462977Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3463191Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3463195Z 
2025-12-04T14:26:42.3463282Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3463344Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3463406Z ====================== 1 failed, 19 deselected in 10.98s =======================
2025-12-04T14:26:42.3463443Z Got exit code 1
2025-12-04T14:26:42.3463482Z Retrying single test...
2025-12-04T14:26:42.3463683Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-e0b85c8a334a8a3b.xml
2025-12-04T14:26:42.3463751Z ============================= test session starts ==============================
2025-12-04T14:26:42.3463863Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3463903Z cachedir: .pytest_cache
2025-12-04T14:26:42.3464060Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3464129Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3464170Z configfile: pytest.ini
2025-12-04T14:26:42.3464330Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3464404Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3464617Z stepcurrent: skipping 17 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3464661Z Running 1 items in this shard
2025-12-04T14:26:42.3464663Z 
2025-12-04T14:26:42.3464959Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda I1204 14:25:17.413000 399170 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 399239
2025-12-04T14:26:42.3465116Z I1204 14:25:17.414000 399170 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 399240
2025-12-04T14:26:42.3465268Z I1204 14:25:17.414000 399170 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 399241
2025-12-04T14:26:42.3465418Z I1204 14:25:17.415000 399170 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 399242
2025-12-04T14:26:42.3465782Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3465830Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3466185Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3466231Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3466583Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3466629Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3467015Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3467061Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3467636Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3467674Z   _warn_cpu_init()
2025-12-04T14:26:42.3468266Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3468318Z   _warn_cpu_init()
2025-12-04T14:26:42.3468897Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3468948Z   _warn_cpu_init()
2025-12-04T14:26:42.3469510Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3469547Z   _warn_cpu_init()
2025-12-04T14:26:42.3469837Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3469879Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3470021Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3470184Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3470472Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3470628Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3470915Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3471040Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3471317Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3471466Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3471741Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3471889Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3472177Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3472314Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3472602Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3472750Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3473244Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3473361Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3473556Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3473903Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3474017Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3474229Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3474395Z [rank3]:E1204 14:25:26.369000 399242 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3474434Z dist init r=3, world=4
2025-12-04T14:26:42.3474572Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3474733Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3475021Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3475176Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3475464Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3475588Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3475866Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3476014Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3476290Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3476458Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3476734Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3476926Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3477219Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3477381Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3477850Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 1. CUDA driver allocated memory was 2317352960 and is now 3919577088.
2025-12-04T14:26:42.3477966Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3478162Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3478507Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3478618Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3478830Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3478996Z [rank1]:E1204 14:25:26.392000 399240 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3479033Z dist init r=1, world=4
2025-12-04T14:26:42.3479171Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3479331Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3479618Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3479772Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3480059Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3480182Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3480460Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3480609Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3480897Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3481057Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3481344Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3481492Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3481771Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3481919Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3482391Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 295424 on device 2. CUDA driver allocated memory was 2300575744 and is now 3902799872.
2025-12-04T14:26:42.3482506Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3482701Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3483046Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3483158Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3483369Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3483534Z [rank2]:E1204 14:25:26.394000 399241 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3483572Z dist init r=2, world=4
2025-12-04T14:26:42.3483709Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3483870Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3484156Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3484310Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3484598Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3484722Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3485008Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3485156Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3485442Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3485614Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3485891Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3486028Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3486306Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3486455Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3486975Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 295424 on device 0. CUDA driver allocated memory was 2453667840 and is now 4055891968.
2025-12-04T14:26:42.3487092Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3487287Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3487631Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3487744Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3487956Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3488120Z [rank0]:E1204 14:25:26.467000 399239 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3488158Z dist init r=0, world=4
2025-12-04T14:26:42.3488495Z [rank0]:[W1204 14:25:26.347868224 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3488536Z FAILED [11.0158s] [100%]
2025-12-04T14:26:42.3488538Z 
2025-12-04T14:26:42.3488594Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3488693Z ________ TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda ________
2025-12-04T14:26:42.3488739Z Traceback (most recent call last):
2025-12-04T14:26:42.3488900Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3488944Z     self._join_processes(fn)
2025-12-04T14:26:42.3489132Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3489186Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3489366Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3489424Z     raise RuntimeError(error)
2025-12-04T14:26:42.3489503Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3489546Z Traceback (most recent call last):
2025-12-04T14:26:42.3489742Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3489784Z     getattr(self, test_name)()
2025-12-04T14:26:42.3489941Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3489975Z     fn()
2025-12-04T14:26:42.3490126Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3490166Z     method(*args, **kwargs)
2025-12-04T14:26:42.3490316Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3490356Z     method(*args, **kwargs)
2025-12-04T14:26:42.3490506Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3490541Z     with policy():
2025-12-04T14:26:42.3490696Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3490735Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3491078Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 1. CUDA driver allocated memory was 2317352960 and is now 3919577088.
2025-12-04T14:26:42.3491080Z 
2025-12-04T14:26:42.3491154Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3491371Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3491374Z 
2025-12-04T14:26:42.3491461Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3491463Z 
2025-12-04T14:26:42.3491523Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3491569Z Traceback (most recent call last):
2025-12-04T14:26:42.3491730Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3491772Z     getattr(self, test_name)()
2025-12-04T14:26:42.3491928Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3491964Z     fn()
2025-12-04T14:26:42.3492113Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3492154Z     method(*args, **kwargs)
2025-12-04T14:26:42.3492303Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3492342Z     method(*args, **kwargs)
2025-12-04T14:26:42.3492491Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3492530Z     with policy():
2025-12-04T14:26:42.3492680Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3492721Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3493071Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3493084Z 
2025-12-04T14:26:42.3493157Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3493373Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3493375Z 
2025-12-04T14:26:42.3493487Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3493489Z 
2025-12-04T14:26:42.3493491Z 
2025-12-04T14:26:42.3493566Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3493653Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3493890Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-e0b85c8a334a8a3b.xml -
2025-12-04T14:26:42.3493950Z =========================== short test summary info ============================
2025-12-04T14:26:42.3494190Z FAILED [11.0158s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3494235Z Traceback (most recent call last):
2025-12-04T14:26:42.3494399Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3494441Z     getattr(self, test_name)()
2025-12-04T14:26:42.3494600Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3494637Z     fn()
2025-12-04T14:26:42.3494786Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3494827Z     method(*args, **kwargs)
2025-12-04T14:26:42.3494976Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3495016Z     method(*args, **kwargs)
2025-12-04T14:26:42.3495165Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3495201Z     with policy():
2025-12-04T14:26:42.3495351Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3495393Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3495732Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 1. CUDA driver allocated memory was 2317352960 and is now 3919577088.
2025-12-04T14:26:42.3495734Z 
2025-12-04T14:26:42.3495809Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3496025Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3496028Z 
2025-12-04T14:26:42.3496115Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3496117Z 
2025-12-04T14:26:42.3496177Z Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3496223Z Traceback (most recent call last):
2025-12-04T14:26:42.3496385Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3496426Z     getattr(self, test_name)()
2025-12-04T14:26:42.3496583Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3496616Z     fn()
2025-12-04T14:26:42.3496831Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3496883Z     method(*args, **kwargs)
2025-12-04T14:26:42.3497032Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3497070Z     method(*args, **kwargs)
2025-12-04T14:26:42.3497219Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3497287Z     with policy():
2025-12-04T14:26:42.3497438Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3497478Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3497819Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda! Caching allocator allocated memory was 512 and is now reported as 227840 on device 3. CUDA driver allocated memory was 2250244096 and is now 3852468224.
2025-12-04T14:26:42.3497822Z 
2025-12-04T14:26:42.3497895Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3498110Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3498112Z 
2025-12-04T14:26:42.3498199Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3498263Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3498325Z ====================== 1 failed, 19 deselected in 11.18s =======================
2025-12-04T14:26:42.3498361Z Got exit code 1
2025-12-04T14:26:42.3498530Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda
2025-12-04T14:26:42.3498658Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.3498846Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-cdbc94eeba4979d7.xml
2025-12-04T14:26:42.3498904Z ============================= test session starts ==============================
2025-12-04T14:26:42.3499015Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3499057Z cachedir: .pytest_cache
2025-12-04T14:26:42.3499214Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3499259Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3499299Z configfile: pytest.ini
2025-12-04T14:26:42.3499460Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3499535Z collecting ... collected 60 items / 18 deselected / 42 selected
2025-12-04T14:26:42.3499587Z stepcurrent: skipping 18 already run items.
2025-12-04T14:26:42.3499632Z Running 2 items in this shard
2025-12-04T14:26:42.3499634Z 
2025-12-04T14:26:42.3499936Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda I1204 14:25:30.946000 399572 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 399641
2025-12-04T14:26:42.3500092Z I1204 14:25:30.947000 399572 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 399642
2025-12-04T14:26:42.3500245Z I1204 14:25:30.948000 399572 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 399643
2025-12-04T14:26:42.3500394Z I1204 14:25:30.948000 399572 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 399644
2025-12-04T14:26:42.3500766Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3500827Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3501188Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3501247Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3501595Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3501641Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3501989Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3502037Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3502322Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3502367Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3502941Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3502980Z   _warn_cpu_init()
2025-12-04T14:26:42.3503259Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3503303Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3503871Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3503908Z   _warn_cpu_init()
2025-12-04T14:26:42.3504194Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3504233Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3504519Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3504558Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3504849Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3504890Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3505178Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3505231Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3505822Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3505872Z   _warn_cpu_init()
2025-12-04T14:26:42.3506148Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3506191Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3506799Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3506838Z   _warn_cpu_init()
2025-12-04T14:26:42.3507123Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3507161Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3507444Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3507482Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3507710Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3507751Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3507979Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3508019Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3508242Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3508283Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3508505Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3508547Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3508765Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3508808Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3509026Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3509068Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3509306Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3509349Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3509566Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3509620Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3509764Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3509960Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3510251Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3510408Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3510695Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3510820Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3511098Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3511246Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3511523Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3511670Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3511945Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3512084Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3512362Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3512512Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3512989Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 1. CUDA driver allocated memory was 2317352960 and is now 3921674240.
2025-12-04T14:26:42.3513107Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3513302Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3513662Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3513778Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3514003Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3514178Z [rank1]:E1204 14:25:40.372000 399642 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3514230Z dist init r=1, world=4
2025-12-04T14:26:42.3514371Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3514530Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3514819Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3514973Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3515259Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3515385Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3515662Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3515809Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3516084Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3516231Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3516507Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3516645Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3516968Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3517117Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3517592Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 0. CUDA driver allocated memory was 2453667840 and is now 4057989120.
2025-12-04T14:26:42.3517707Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3517919Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3518268Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3518396Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3518634Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3518797Z [rank0]:E1204 14:25:40.435000 399641 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3518838Z dist init r=0, world=4
2025-12-04T14:26:42.3518977Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3519136Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3519423Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3519580Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3519866Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3519990Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3520266Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3520413Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3520691Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3520837Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3521114Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3521250Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3521527Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3521678Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3522162Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1158656 on device 3. CUDA driver allocated memory was 2250244096 and is now 3854565376.
2025-12-04T14:26:42.3522277Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3522488Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3522847Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3522970Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3523183Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3523348Z [rank3]:E1204 14:25:40.463000 399644 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3523388Z dist init r=3, world=4
2025-12-04T14:26:42.3523526Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3523684Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3523972Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3524126Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3524417Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3524541Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3524819Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3524968Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3525244Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3525392Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3525669Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3525807Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3526085Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3526233Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3526715Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 2. CUDA driver allocated memory was 2300575744 and is now 3904897024.
2025-12-04T14:26:42.3526886Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3527108Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3527457Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3527569Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3527780Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3527943Z [rank2]:E1204 14:25:40.470000 399643 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3527982Z dist init r=2, world=4
2025-12-04T14:26:42.3528318Z [rank0]:[W1204 14:25:40.291867904 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3528358Z FAILED [11.4161s] [ 50%]
2025-12-04T14:26:42.3528361Z 
2025-12-04T14:26:42.3528416Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3528514Z ______ TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda _______
2025-12-04T14:26:42.3528560Z Traceback (most recent call last):
2025-12-04T14:26:42.3528723Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3528766Z     self._join_processes(fn)
2025-12-04T14:26:42.3528940Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3528995Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3529173Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3529215Z     raise RuntimeError(error)
2025-12-04T14:26:42.3529294Z RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3529339Z Traceback (most recent call last):
2025-12-04T14:26:42.3529500Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3529543Z     getattr(self, test_name)()
2025-12-04T14:26:42.3529700Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3529735Z     fn()
2025-12-04T14:26:42.3529887Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3529928Z     method(*args, **kwargs)
2025-12-04T14:26:42.3530078Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3530118Z     method(*args, **kwargs)
2025-12-04T14:26:42.3530266Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3530316Z     with policy():
2025-12-04T14:26:42.3530468Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3530521Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3530867Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 1. CUDA driver allocated memory was 2317352960 and is now 3921674240.
2025-12-04T14:26:42.3530895Z 
2025-12-04T14:26:42.3530971Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3531193Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3531195Z 
2025-12-04T14:26:42.3531285Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3531287Z 
2025-12-04T14:26:42.3531289Z 
2025-12-04T14:26:42.3531362Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3531452Z Process 1 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3531689Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-cdbc94eeba4979d7.xml -
2025-12-04T14:26:42.3531749Z =========================== short test summary info ============================
2025-12-04T14:26:42.3531993Z FAILED [11.4161s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda - RuntimeError: Process 1 exited with error code 10 and exception:
2025-12-04T14:26:42.3532038Z Traceback (most recent call last):
2025-12-04T14:26:42.3532203Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3532244Z     getattr(self, test_name)()
2025-12-04T14:26:42.3532403Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3532438Z     fn()
2025-12-04T14:26:42.3532589Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3532628Z     method(*args, **kwargs)
2025-12-04T14:26:42.3532779Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3532818Z     method(*args, **kwargs)
2025-12-04T14:26:42.3532968Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3533003Z     with policy():
2025-12-04T14:26:42.3533157Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3533196Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3533546Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 1. CUDA driver allocated memory was 2317352960 and is now 3921674240.
2025-12-04T14:26:42.3533549Z 
2025-12-04T14:26:42.3533623Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3533846Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3533848Z 
2025-12-04T14:26:42.3533935Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3533996Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3534071Z ====================== 1 failed, 18 deselected in 11.58s =======================
2025-12-04T14:26:42.3534108Z Got exit code 1
2025-12-04T14:26:42.3534148Z Retrying single test...
2025-12-04T14:26:42.3534350Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-764a914dc6fca497.xml
2025-12-04T14:26:42.3534407Z ============================= test session starts ==============================
2025-12-04T14:26:42.3534518Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3534582Z cachedir: .pytest_cache
2025-12-04T14:26:42.3534740Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3534787Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3534827Z configfile: pytest.ini
2025-12-04T14:26:42.3534990Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3535064Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3535281Z stepcurrent: skipping 18 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3535326Z Running 1 items in this shard
2025-12-04T14:26:42.3535329Z 
2025-12-04T14:26:42.3535633Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda I1204 14:25:44.923000 399974 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 400043
2025-12-04T14:26:42.3535791Z I1204 14:25:44.924000 399974 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 400044
2025-12-04T14:26:42.3535943Z I1204 14:25:44.925000 399974 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 400045
2025-12-04T14:26:42.3536095Z I1204 14:25:44.925000 399974 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 400046
2025-12-04T14:26:42.3536456Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3536507Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3536906Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3536956Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3537310Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3537356Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3537705Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3537750Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3538034Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3538077Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3538674Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3538725Z   _warn_cpu_init()
2025-12-04T14:26:42.3539026Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3539079Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3539366Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3539409Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3539684Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3539728Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3540298Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3540338Z   _warn_cpu_init()
2025-12-04T14:26:42.3540617Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3540658Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3541229Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3541267Z   _warn_cpu_init()
2025-12-04T14:26:42.3541543Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3541584Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3542152Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3542190Z   _warn_cpu_init()
2025-12-04T14:26:42.3542475Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3542514Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3542809Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3542848Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3543132Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3543182Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3543423Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3543480Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3543704Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3543746Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3543967Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3544009Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3544230Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3544269Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3544488Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3544528Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3544748Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3544787Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3545006Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3545045Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3545264Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3545303Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3545453Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3545616Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3545907Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3546063Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3546350Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3546477Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3546791Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3546955Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3547232Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3547395Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3547684Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3547834Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3548114Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3548261Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3548737Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 3. CUDA driver allocated memory was 2250244096 and is now 3854565376.
2025-12-04T14:26:42.3548853Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3549049Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3549399Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3549512Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3549725Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3549891Z [rank3]:E1204 14:25:54.306000 400046 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3549930Z dist init r=3, world=4
2025-12-04T14:26:42.3550069Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3550230Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3550518Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3550671Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3550959Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3551081Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3551372Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3551529Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3551817Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3551986Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3552261Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3552398Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3552679Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3552827Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3553301Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 2. CUDA driver allocated memory was 2300575744 and is now 3904897024.
2025-12-04T14:26:42.3553416Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3553610Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3555092Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3555481Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3555841Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3556121Z [rank2]:E1204 14:25:54.358000 400045 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3556188Z dist init r=2, world=4
2025-12-04T14:26:42.3556438Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3556701Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3557243Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3557493Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3558391Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3558599Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3559118Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3559497Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3559941Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3560184Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3560622Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3560851Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3561302Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3561542Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3562311Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 0. CUDA driver allocated memory was 2453667840 and is now 4057989120.
2025-12-04T14:26:42.3562500Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3562823Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3563391Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3563577Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3563923Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3564215Z [rank0]:E1204 14:25:54.367000 400043 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3564284Z dist init r=0, world=4
2025-12-04T14:26:42.3564509Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3564775Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3565245Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3565517Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3565992Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3566197Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3566682Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3566978Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3567389Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3567571Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3567914Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3568083Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3568433Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3568622Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3569205Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 1. CUDA driver allocated memory was 2317352960 and is now 3921674240.
2025-12-04T14:26:42.3569355Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3569596Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3570031Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3570171Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3570434Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3570641Z [rank1]:E1204 14:25:54.374000 400044 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3570690Z dist init r=1, world=4
2025-12-04T14:26:42.3572669Z [rank0]:[W1204 14:25:54.158136755 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3573013Z FAILED [11.3174s] [100%]
2025-12-04T14:26:42.3573023Z 
2025-12-04T14:26:42.3573209Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3573570Z ______ TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda _______
2025-12-04T14:26:42.3573688Z Traceback (most recent call last):
2025-12-04T14:26:42.3574165Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3574469Z     self._join_processes(fn)
2025-12-04T14:26:42.3574910Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3575055Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3575490Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3575613Z     raise RuntimeError(error)
2025-12-04T14:26:42.3575820Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3575940Z Traceback (most recent call last):
2025-12-04T14:26:42.3576346Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3576452Z     getattr(self, test_name)()
2025-12-04T14:26:42.3576871Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3576957Z     fn()
2025-12-04T14:26:42.3577292Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3577385Z     method(*args, **kwargs)
2025-12-04T14:26:42.3577722Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3577816Z     method(*args, **kwargs)
2025-12-04T14:26:42.3578143Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3578234Z     with policy():
2025-12-04T14:26:42.3578569Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3578659Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3579460Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 3. CUDA driver allocated memory was 2250244096 and is now 3854565376.
2025-12-04T14:26:42.3579467Z 
2025-12-04T14:26:42.3579634Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3580145Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3580149Z 
2025-12-04T14:26:42.3580351Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3580363Z 
2025-12-04T14:26:42.3580370Z 
2025-12-04T14:26:42.3580548Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3580746Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3581273Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-764a914dc6fca497.xml -
2025-12-04T14:26:42.3581416Z =========================== short test summary info ============================
2025-12-04T14:26:42.3581993Z FAILED [11.3174s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3582137Z Traceback (most recent call last):
2025-12-04T14:26:42.3582504Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3582637Z     getattr(self, test_name)()
2025-12-04T14:26:42.3582991Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3583068Z     fn()
2025-12-04T14:26:42.3583428Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3583560Z     method(*args, **kwargs)
2025-12-04T14:26:42.3583885Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3583977Z     method(*args, **kwargs)
2025-12-04T14:26:42.3584301Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3584391Z     with policy():
2025-12-04T14:26:42.3584721Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3584814Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3585589Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 3. CUDA driver allocated memory was 2250244096 and is now 3854565376.
2025-12-04T14:26:42.3585603Z 
2025-12-04T14:26:42.3585767Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3586265Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3586271Z 
2025-12-04T14:26:42.3586461Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3586605Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3586832Z ====================== 1 failed, 19 deselected in 11.47s =======================
2025-12-04T14:26:42.3586924Z Got exit code 1
2025-12-04T14:26:42.3587012Z Retrying single test...
2025-12-04T14:26:42.3587417Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1b7b6b5389700b1d.xml
2025-12-04T14:26:42.3587532Z ============================= test session starts ==============================
2025-12-04T14:26:42.3587741Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3587806Z cachedir: .pytest_cache
2025-12-04T14:26:42.3588052Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3588124Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3588191Z configfile: pytest.ini
2025-12-04T14:26:42.3588444Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3588561Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3588893Z stepcurrent: skipping 18 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3588963Z Running 1 items in this shard
2025-12-04T14:26:42.3588968Z 
2025-12-04T14:26:42.3589433Z distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda I1204 14:25:58.928000 400376 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 400445
2025-12-04T14:26:42.3589669Z I1204 14:25:58.929000 400376 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 400446
2025-12-04T14:26:42.3589931Z I1204 14:25:58.930000 400376 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 400447
2025-12-04T14:26:42.3590182Z I1204 14:25:58.930000 400376 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 400448
2025-12-04T14:26:42.3590760Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3590861Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3591406Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3591482Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3592013Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3592088Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3592619Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3592694Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3593124Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3593199Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3594086Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3594148Z   _warn_cpu_init()
2025-12-04T14:26:42.3594576Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3594643Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3595514Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3595574Z   _warn_cpu_init()
2025-12-04T14:26:42.3595998Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3596068Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3596997Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3597079Z   _warn_cpu_init()
2025-12-04T14:26:42.3597449Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/wrap.py:91: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3597535Z   return fsdp_fn(module, **kwargs)
2025-12-04T14:26:42.3598216Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:1014: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2025-12-04T14:26:42.3598265Z   _warn_cpu_init()
2025-12-04T14:26:42.3598608Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3598657Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3598997Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3599045Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3599378Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3599424Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3599757Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_fsdp.py:395: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.
2025-12-04T14:26:42.3599804Z   fsdp_model = FSDP(
2025-12-04T14:26:42.3600148Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
2025-12-04T14:26:42.3600202Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3600475Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3600529Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3600790Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3600843Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3601102Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3601154Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3601415Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict will be returned.
2025-12-04T14:26:42.3601469Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3601725Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3601779Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3602051Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3602104Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3602373Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3602425Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3602692Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py:124: UserWarning: When using ``NO_SHARD`` for ``ShardingStrategy``, full_state_dict willbe returned.
2025-12-04T14:26:42.3602759Z   return func(*args, **kwargs)
2025-12-04T14:26:42.3602934Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3603132Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3603477Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3603663Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3604002Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3604155Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3604486Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3604662Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3604991Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3605170Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3605493Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3605658Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3605984Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3606162Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3606723Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 0. CUDA driver allocated memory was 2453667840 and is now 4057989120.
2025-12-04T14:26:42.3607108Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3607363Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3607761Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3607895Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3608126Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3608297Z [rank0]:E1204 14:26:08.296000 400445 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3608339Z dist init r=0, world=4
2025-12-04T14:26:42.3608485Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3608647Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3608939Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3609101Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3609387Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3609519Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3609798Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3609952Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3610232Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3610383Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3610664Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3610803Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3611084Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3611235Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3611725Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 2. CUDA driver allocated memory was 2300575744 and is now 3904897024.
2025-12-04T14:26:42.3611857Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3612057Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3612423Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3612550Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3612769Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3612935Z [rank2]:E1204 14:26:08.306000 400447 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3612981Z dist init r=2, world=4
2025-12-04T14:26:42.3613120Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3613286Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3613578Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3613734Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3614025Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3614152Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3614437Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3614587Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3614869Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3615018Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3615299Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3615442Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3615721Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3615888Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3616361Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 3. CUDA driver allocated memory was 2250244096 and is now 3854565376.
2025-12-04T14:26:42.3616502Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3616728Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3617143Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3617262Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3617477Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3617646Z [rank3]:E1204 14:26:08.306000 400448 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3617688Z dist init r=3, world=4
2025-12-04T14:26:42.3617833Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3617995Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3618289Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3618450Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3618738Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3618871Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3619151Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3619306Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3619584Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3619737Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3620015Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3620156Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3620457Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3620635Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3621122Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 1. CUDA driver allocated memory was 2317352960 and is now 3921674240.
2025-12-04T14:26:42.3621252Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3621472Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3621827Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3621942Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3622161Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3622327Z [rank1]:E1204 14:26:08.336000 400446 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3622371Z dist init r=1, world=4
2025-12-04T14:26:42.3622734Z [rank0]:[W1204 14:26:08.986656016 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-12-04T14:26:42.3622784Z FAILED [11.3181s] [100%]
2025-12-04T14:26:42.3622787Z 
2025-12-04T14:26:42.3622847Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3622951Z ______ TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda _______
2025-12-04T14:26:42.3623002Z Traceback (most recent call last):
2025-12-04T14:26:42.3623171Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3623219Z     self._join_processes(fn)
2025-12-04T14:26:42.3623397Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3623457Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3623638Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3623688Z     raise RuntimeError(error)
2025-12-04T14:26:42.3623772Z RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3623822Z Traceback (most recent call last):
2025-12-04T14:26:42.3623985Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3624035Z     getattr(self, test_name)()
2025-12-04T14:26:42.3624194Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3624237Z     fn()
2025-12-04T14:26:42.3624391Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3624442Z     method(*args, **kwargs)
2025-12-04T14:26:42.3624609Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3624668Z     method(*args, **kwargs)
2025-12-04T14:26:42.3624819Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3624863Z     with policy():
2025-12-04T14:26:42.3625015Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3625087Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3625436Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 0. CUDA driver allocated memory was 2453667840 and is now 4057989120.
2025-12-04T14:26:42.3625443Z 
2025-12-04T14:26:42.3625522Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3625750Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3625753Z 
2025-12-04T14:26:42.3625843Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3625845Z 
2025-12-04T14:26:42.3625847Z 
2025-12-04T14:26:42.3625930Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3626021Z Process 0 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3626263Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-1b7b6b5389700b1d.xml -
2025-12-04T14:26:42.3626327Z =========================== short test summary info ============================
2025-12-04T14:26:42.3626576Z FAILED [11.3181s] distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda - RuntimeError: Process 0 exited with error code 10 and exception:
2025-12-04T14:26:42.3626625Z Traceback (most recent call last):
2025-12-04T14:26:42.3626832Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3626879Z     getattr(self, test_name)()
2025-12-04T14:26:42.3627044Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3627084Z     fn()
2025-12-04T14:26:42.3627241Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3627287Z     method(*args, **kwargs)
2025-12-04T14:26:42.3627440Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3627487Z     method(*args, **kwargs)
2025-12-04T14:26:42.3627640Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3627684Z     with policy():
2025-12-04T14:26:42.3627839Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3627885Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3628236Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda! Caching allocator allocated memory was 512 and is now reported as 1091072 on device 0. CUDA driver allocated memory was 2453667840 and is now 4057989120.
2025-12-04T14:26:42.3628239Z 
2025-12-04T14:26:42.3628319Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3628561Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParityWithDDPCUDA.test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3628563Z 
2025-12-04T14:26:42.3628658Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3628737Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3628807Z ====================== 1 failed, 19 deselected in 11.48s =======================
2025-12-04T14:26:42.3634152Z Got exit code 1
2025-12-04T14:26:42.3634568Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda
2025-12-04T14:26:42.3634725Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.3634921Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-f341a13c377d3500.xml
2025-12-04T14:26:42.3634984Z ============================= test session starts ==============================
2025-12-04T14:26:42.3635108Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3635152Z cachedir: .pytest_cache
2025-12-04T14:26:42.3635320Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3635370Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3635417Z configfile: pytest.ini
2025-12-04T14:26:42.3635582Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3635666Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3635723Z stepcurrent: skipping 19 already run items.
2025-12-04T14:26:42.3635773Z Running 1 items in this shard
2025-12-04T14:26:42.3635775Z 
2025-12-04T14:26:42.3636092Z distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda I1204 14:26:12.688000 400778 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 400847
2025-12-04T14:26:42.3636256Z I1204 14:26:12.689000 400778 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 400848
2025-12-04T14:26:42.3636412Z I1204 14:26:12.690000 400778 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 400849
2025-12-04T14:26:42.3636569Z I1204 14:26:12.690000 400778 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 400850
2025-12-04T14:26:42.3636987Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3637040Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3637340Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3637412Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3637523Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3637600Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3638105Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3638171Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3638569Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3638638Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3638941Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3639025Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3639131Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3639209Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3639700Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3639767Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3640123Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3640177Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3640472Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3640538Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3640646Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3640721Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3641211Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3641273Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3641637Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3641687Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3641980Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3642049Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3642152Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3642231Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3642741Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3642806Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3642969Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3643139Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3643447Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3643623Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3643917Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3644046Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3644333Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3644486Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3644769Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3644920Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3645202Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3645346Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3645627Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3645782Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3646266Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 2. CUDA driver allocated memory was 2300575744 and is now 3105882112.
2025-12-04T14:26:42.3646390Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3646591Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3647007Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3647141Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3647358Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3647541Z [rank2]:E1204 14:26:18.880000 400849 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3647583Z dist init r=2, world=4
2025-12-04T14:26:42.3647738Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3647916Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3648210Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3648366Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3648657Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3648788Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3649071Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3649225Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3649505Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3649658Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3649936Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3650079Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3650365Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3650514Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3650997Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 0. CUDA driver allocated memory was 2453667840 and is now 3258974208.
2025-12-04T14:26:42.3651115Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3651319Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3651692Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3651822Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3652041Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3652240Z [rank0]:E1204 14:26:18.890000 400847 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3652286Z dist init r=0, world=4
2025-12-04T14:26:42.3652426Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3652593Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3652882Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3653043Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3653331Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3653461Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3653749Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3653899Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3654182Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3654332Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3654615Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3654753Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3655038Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3655192Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3655670Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 3. CUDA driver allocated memory was 2250244096 and is now 3055550464.
2025-12-04T14:26:42.3655808Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3656007Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3656381Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3656524Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3656794Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3656967Z [rank3]:E1204 14:26:18.893000 400850 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3657010Z dist init r=3, world=4
2025-12-04T14:26:42.3657155Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3657319Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3657614Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3657770Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3658064Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3658190Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3658475Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3658629Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3658909Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3659061Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3659340Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3659483Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3659764Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3659919Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3660418Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 1. CUDA driver allocated memory was 2317352960 and is now 3122659328.
2025-12-04T14:26:42.3660547Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3660751Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3661145Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3661264Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3661478Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3661649Z [rank1]:E1204 14:26:18.923000 400848 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3661693Z dist init r=1, world=4
2025-12-04T14:26:42.3661734Z FAILED [7.3131s] [100%]
2025-12-04T14:26:42.3661737Z 
2025-12-04T14:26:42.3661799Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3661903Z ___ TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda ___
2025-12-04T14:26:42.3661956Z Traceback (most recent call last):
2025-12-04T14:26:42.3662120Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3662170Z     self._join_processes(fn)
2025-12-04T14:26:42.3662346Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3662406Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3662587Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3662638Z     raise RuntimeError(error)
2025-12-04T14:26:42.3662721Z RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.3662774Z Traceback (most recent call last):
2025-12-04T14:26:42.3662938Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3662988Z     getattr(self, test_name)()
2025-12-04T14:26:42.3663149Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3663191Z     fn()
2025-12-04T14:26:42.3663346Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3663397Z     method(*args, **kwargs)
2025-12-04T14:26:42.3663550Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3663596Z     method(*args, **kwargs)
2025-12-04T14:26:42.3663747Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3663793Z     with policy():
2025-12-04T14:26:42.3663951Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3663995Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3664364Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 2. CUDA driver allocated memory was 2300575744 and is now 3105882112.
2025-12-04T14:26:42.3664370Z 
2025-12-04T14:26:42.3664458Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3664696Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3664698Z 
2025-12-04T14:26:42.3664789Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3664811Z 
2025-12-04T14:26:42.3664813Z 
2025-12-04T14:26:42.3664897Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3664988Z Process 2 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3665231Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-f341a13c377d3500.xml -
2025-12-04T14:26:42.3665297Z =========================== short test summary info ============================
2025-12-04T14:26:42.3665548Z FAILED [7.3131s] distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda - RuntimeError: Process 2 exited with error code 10 and exception:
2025-12-04T14:26:42.3665602Z Traceback (most recent call last):
2025-12-04T14:26:42.3665769Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3665820Z     getattr(self, test_name)()
2025-12-04T14:26:42.3665981Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3666023Z     fn()
2025-12-04T14:26:42.3666176Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3666222Z     method(*args, **kwargs)
2025-12-04T14:26:42.3666375Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3666421Z     method(*args, **kwargs)
2025-12-04T14:26:42.3666572Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3666615Z     with policy():
2025-12-04T14:26:42.3666819Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3666868Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3667220Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 2. CUDA driver allocated memory was 2300575744 and is now 3105882112.
2025-12-04T14:26:42.3667226Z 
2025-12-04T14:26:42.3667303Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3667539Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3667542Z 
2025-12-04T14:26:42.3667631Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3667701Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3667769Z ======================= 1 failed, 19 deselected in 7.47s =======================
2025-12-04T14:26:42.3667812Z Got exit code 1
2025-12-04T14:26:42.3667855Z Retrying single test...
2025-12-04T14:26:42.3668053Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-7a8f748700be72d0.xml
2025-12-04T14:26:42.3668114Z ============================= test session starts ==============================
2025-12-04T14:26:42.3668246Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3668291Z cachedir: .pytest_cache
2025-12-04T14:26:42.3668470Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3668520Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3668567Z configfile: pytest.ini
2025-12-04T14:26:42.3668753Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3668849Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3669075Z stepcurrent: skipping 19 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3669127Z Running 1 items in this shard
2025-12-04T14:26:42.3669129Z 
2025-12-04T14:26:42.3669442Z distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda I1204 14:26:22.398000 401156 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 401225
2025-12-04T14:26:42.3669602Z I1204 14:26:22.398000 401156 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 401226
2025-12-04T14:26:42.3669760Z I1204 14:26:22.399000 401156 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 401227
2025-12-04T14:26:42.3669914Z I1204 14:26:22.399000 401156 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 401228
2025-12-04T14:26:42.3670281Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3670334Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3670693Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3670743Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3671040Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3671113Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3671220Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3671299Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3671792Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3671860Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3672151Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3672222Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3672326Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3672405Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3672914Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3672985Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3673357Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3673417Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3673710Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3673774Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3673883Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3673957Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3674448Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3674514Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3674869Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3674923Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3675212Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3675280Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3675384Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3675462Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3675949Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3676014Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3676166Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3676331Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3676628Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3676836Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3678049Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3678492Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3678953Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3679217Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3679578Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3679779Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3680128Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3680303Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3680653Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3680847Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3681455Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 3. CUDA driver allocated memory was 2250244096 and is now 3055550464.
2025-12-04T14:26:42.3681609Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3681868Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3682329Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3682478Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3682743Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3682955Z [rank3]:E1204 14:26:28.482000 401228 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3683011Z dist init r=3, world=4
2025-12-04T14:26:42.3683198Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3683401Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3683775Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3683985Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3684342Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3684533Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3684884Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3685079Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3685418Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3685611Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3685964Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3686132Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3686496Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3686658Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3687246Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 0. CUDA driver allocated memory was 2453667840 and is now 3258974208.
2025-12-04T14:26:42.3687377Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3687594Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3687987Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3688114Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3688348Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3688530Z [rank0]:E1204 14:26:28.493000 401225 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3688579Z dist init r=0, world=4
2025-12-04T14:26:42.3688746Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3688928Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3689259Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3689445Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3689774Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3689911Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3690218Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3690380Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3690689Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3690854Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3691160Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3691315Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3691619Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3691790Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3692310Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 2. CUDA driver allocated memory was 2300575744 and is now 3105882112.
2025-12-04T14:26:42.3692439Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3692659Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3693050Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3693182Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3693412Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3693608Z [rank2]:E1204 14:26:28.504000 401227 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3693663Z dist init r=2, world=4
2025-12-04T14:26:42.3693819Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3693996Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3694352Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3694523Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3694834Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3694976Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3695279Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3695452Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3695755Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3695918Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3696217Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3696368Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3696666Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3696863Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3697341Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 1. CUDA driver allocated memory was 2317352960 and is now 3122659328.
2025-12-04T14:26:42.3697456Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3697654Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3698012Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3698141Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3698355Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3698537Z [rank1]:E1204 14:26:28.514000 401226 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3698576Z dist init r=1, world=4
2025-12-04T14:26:42.3698632Z FAILED [7.0126s] [100%]
2025-12-04T14:26:42.3698651Z 
2025-12-04T14:26:42.3698729Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3698835Z ___ TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda ___
2025-12-04T14:26:42.3698887Z Traceback (most recent call last):
2025-12-04T14:26:42.3699058Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3699104Z     self._join_processes(fn)
2025-12-04T14:26:42.3699282Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3699340Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3699522Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3699571Z     raise RuntimeError(error)
2025-12-04T14:26:42.3699656Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3699709Z Traceback (most recent call last):
2025-12-04T14:26:42.3699873Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3699921Z     getattr(self, test_name)()
2025-12-04T14:26:42.3700081Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3700122Z     fn()
2025-12-04T14:26:42.3700274Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3700322Z     method(*args, **kwargs)
2025-12-04T14:26:42.3700475Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3700522Z     method(*args, **kwargs)
2025-12-04T14:26:42.3700674Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3700717Z     with policy():
2025-12-04T14:26:42.3700869Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3700914Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3701268Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 3. CUDA driver allocated memory was 2250244096 and is now 3055550464.
2025-12-04T14:26:42.3701272Z 
2025-12-04T14:26:42.3701353Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3701589Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3701598Z 
2025-12-04T14:26:42.3701689Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3701692Z 
2025-12-04T14:26:42.3701694Z 
2025-12-04T14:26:42.3701778Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3701867Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3702128Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-7a8f748700be72d0.xml -
2025-12-04T14:26:42.3702202Z =========================== short test summary info ============================
2025-12-04T14:26:42.3702497Z FAILED [7.0126s] distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3702545Z Traceback (most recent call last):
2025-12-04T14:26:42.3702737Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3702782Z     getattr(self, test_name)()
2025-12-04T14:26:42.3702947Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3702984Z     fn()
2025-12-04T14:26:42.3703140Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3703181Z     method(*args, **kwargs)
2025-12-04T14:26:42.3703337Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3703381Z     method(*args, **kwargs)
2025-12-04T14:26:42.3703532Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3703575Z     with policy():
2025-12-04T14:26:42.3703729Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3703775Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3704131Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 3. CUDA driver allocated memory was 2250244096 and is now 3055550464.
2025-12-04T14:26:42.3704133Z 
2025-12-04T14:26:42.3704211Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3704443Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3704445Z 
2025-12-04T14:26:42.3704537Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3704603Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3704670Z ======================= 1 failed, 19 deselected in 7.17s =======================
2025-12-04T14:26:42.3704708Z Got exit code 1
2025-12-04T14:26:42.3704753Z Retrying single test...
2025-12-04T14:26:42.3704947Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-ca636b967da8e848.xml
2025-12-04T14:26:42.3705012Z ============================= test session starts ==============================
2025-12-04T14:26:42.3705127Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3705174Z cachedir: .pytest_cache
2025-12-04T14:26:42.3705338Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3705386Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3705432Z configfile: pytest.ini
2025-12-04T14:26:42.3705599Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3705679Z collecting ... collected 60 items / 19 deselected / 41 selected
2025-12-04T14:26:42.3705905Z stepcurrent: skipping 19 already run items. Running only test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3705968Z Running 1 items in this shard
2025-12-04T14:26:42.3705971Z 
2025-12-04T14:26:42.3706283Z distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda I1204 14:26:31.974000 401534 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 401603
2025-12-04T14:26:42.3706460Z I1204 14:26:31.975000 401534 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 401604
2025-12-04T14:26:42.3706635Z I1204 14:26:31.975000 401534 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 401605
2025-12-04T14:26:42.3706828Z I1204 14:26:31.976000 401534 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 401606
2025-12-04T14:26:42.3707195Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3707247Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3707608Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3707657Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3707957Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3708026Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3708135Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3708213Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3708504Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3708570Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3708677Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3708751Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3709250Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3709315Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3709805Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3709871Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3710231Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3710283Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3710589Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3710675Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3710780Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3710851Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3711371Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3711431Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3711790Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance)
2025-12-04T14:26:42.3711838Z   self.encoder = TransformerEncoder(
2025-12-04T14:26:42.3712129Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_wrap_utils.py:64: UserWarning: Both mixed precision and an auto_wrap_policy were specified to FSDP, where the wrapped module has submodules of type:
2025-12-04T14:26:42.3712192Z {<class 'torch.nn.modules.batchnorm.BatchNorm1d'>}
2025-12-04T14:26:42.3712297Z These modules will be wrapped as separate FSDP instacnes with mixed precision disabled.
2025-12-04T14:26:42.3712372Z   _warn_on_overridden_mixed_precision(overridden_module_classes)
2025-12-04T14:26:42.3712863Z /opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.
2025-12-04T14:26:42.3712925Z   device_from_device_id = _get_device_from_device_id(
2025-12-04T14:26:42.3713072Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3713241Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3713534Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3713696Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3713984Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3714116Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3714400Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3714549Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3714844Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3715005Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3715285Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3715448Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3715730Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3715884Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3716364Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 3. CUDA driver allocated memory was 2250244096 and is now 3055550464.
2025-12-04T14:26:42.3716487Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3716686Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3717276Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3717400Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3717643Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3717813Z [rank3]:E1204 14:26:38.199000 401606 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 3 with exit code: 10
2025-12-04T14:26:42.3717854Z dist init r=3, world=4
2025-12-04T14:26:42.3717997Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3718158Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3718454Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3718610Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3718900Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3719029Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3719322Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3719476Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3719774Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3719939Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3720229Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3720372Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3720654Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3720807Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3721288Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 0. CUDA driver allocated memory was 2453667840 and is now 3258974208.
2025-12-04T14:26:42.3721404Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3721605Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3721962Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3722083Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3722300Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3722466Z [rank0]:E1204 14:26:38.214000 401603 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 0 with exit code: 10
2025-12-04T14:26:42.3722509Z dist init r=0, world=4
2025-12-04T14:26:42.3722648Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3722813Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3723102Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3723261Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3723549Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3723689Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3723971Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3724133Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3724439Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3724588Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3724868Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3725005Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3725289Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3725443Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3725919Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 1. CUDA driver allocated memory was 2317352960 and is now 3122659328.
2025-12-04T14:26:42.3726038Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3726235Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3726595Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3726709Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3726964Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3727135Z [rank1]:E1204 14:26:38.224000 401604 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 1 with exit code: 10
2025-12-04T14:26:42.3727176Z dist init r=1, world=4
2025-12-04T14:26:42.3727318Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 
2025-12-04T14:26:42.3727480Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last):
2025-12-04T14:26:42.3727775Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3727955Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]     getattr(self, test_name)()
2025-12-04T14:26:42.3728247Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3728389Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]     fn()
2025-12-04T14:26:42.3728687Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3728859Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3729200Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3729414Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]     method(*args, **kwargs)
2025-12-04T14:26:42.3729696Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3729835Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]     with policy():
2025-12-04T14:26:42.3730118Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3730269Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]     raise RuntimeError(msg)
2025-12-04T14:26:42.3730752Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 2. CUDA driver allocated memory was 2300575744 and is now 3105882112.
2025-12-04T14:26:42.3730871Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3731069Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3731429Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3731542Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935] 
2025-12-04T14:26:42.3731758Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3731923Z [rank2]:E1204 14:26:38.290000 401605 site-packages/torch/testing/_internal/common_distributed.py:935]  exiting process 2 with exit code: 10
2025-12-04T14:26:42.3731968Z dist init r=2, world=4
2025-12-04T14:26:42.3732008Z FAILED [7.3135s] [100%]
2025-12-04T14:26:42.3732010Z 
2025-12-04T14:26:42.3732072Z =================================== FAILURES ===================================
2025-12-04T14:26:42.3732173Z ___ TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda ___
2025-12-04T14:26:42.3732223Z Traceback (most recent call last):
2025-12-04T14:26:42.3732403Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
2025-12-04T14:26:42.3732467Z     self._join_processes(fn)
2025-12-04T14:26:42.3732645Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
2025-12-04T14:26:42.3732702Z     self._check_return_codes(fn, elapsed_time)
2025-12-04T14:26:42.3732896Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
2025-12-04T14:26:42.3732952Z     raise RuntimeError(error)
2025-12-04T14:26:42.3733036Z RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3733081Z Traceback (most recent call last):
2025-12-04T14:26:42.3733247Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3733292Z     getattr(self, test_name)()
2025-12-04T14:26:42.3733453Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3733491Z     fn()
2025-12-04T14:26:42.3733645Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3733688Z     method(*args, **kwargs)
2025-12-04T14:26:42.3733841Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3733885Z     method(*args, **kwargs)
2025-12-04T14:26:42.3734038Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3734077Z     with policy():
2025-12-04T14:26:42.3734232Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3734276Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3734629Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 3. CUDA driver allocated memory was 2250244096 and is now 3055550464.
2025-12-04T14:26:42.3734633Z 
2025-12-04T14:26:42.3734712Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3734947Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3734951Z 
2025-12-04T14:26:42.3735043Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3735045Z 
2025-12-04T14:26:42.3735047Z 
2025-12-04T14:26:42.3735124Z ----------------------------- Captured stdout call -----------------------------
2025-12-04T14:26:42.3735217Z Process 3 terminated with exit code 10, terminating remaining processes.
2025-12-04T14:26:42.3735452Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-ca636b967da8e848.xml -
2025-12-04T14:26:42.3735517Z =========================== short test summary info ============================
2025-12-04T14:26:42.3735765Z FAILED [7.3135s] distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda - RuntimeError: Process 3 exited with error code 10 and exception:
2025-12-04T14:26:42.3735815Z Traceback (most recent call last):
2025-12-04T14:26:42.3735980Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
2025-12-04T14:26:42.3736027Z     getattr(self, test_name)()
2025-12-04T14:26:42.3736205Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
2025-12-04T14:26:42.3736242Z     fn()
2025-12-04T14:26:42.3736396Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3736448Z     method(*args, **kwargs)
2025-12-04T14:26:42.3736602Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
2025-12-04T14:26:42.3736642Z     method(*args, **kwargs)
2025-12-04T14:26:42.3736862Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper
2025-12-04T14:26:42.3736915Z     with policy():
2025-12-04T14:26:42.3737070Z   File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__
2025-12-04T14:26:42.3737111Z     raise RuntimeError(msg)
2025-12-04T14:26:42.3737470Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda! Caching allocator allocated memory was 512 and is now reported as 28160 on device 3. CUDA driver allocated memory was 2250244096 and is now 3055550464.
2025-12-04T14:26:42.3737473Z 
2025-12-04T14:26:42.3737548Z To execute this test, run the following from the base repo dir:
2025-12-04T14:26:42.3737783Z     PYTORCH_TEST_WITH_ROCM=1 PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_core.py TestParamInitCUDA.test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3737786Z 
2025-12-04T14:26:42.3737906Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
2025-12-04T14:26:42.3737978Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-12-04T14:26:42.3738064Z ======================= 1 failed, 19 deselected in 7.48s =======================
2025-12-04T14:26:42.3738120Z Got exit code 1
2025-12-04T14:26:42.3738327Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda
2025-12-04T14:26:42.3738478Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-12-04T14:26:42.3738670Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-fb3f972c7effba9d.xml
2025-12-04T14:26:42.3738733Z ============================= test session starts ==============================
2025-12-04T14:26:42.3738848Z platform linux -- Python 3.12.5, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.12/bin/python
2025-12-04T14:26:42.3738895Z cachedir: .pytest_cache
2025-12-04T14:26:42.3739054Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-12-04T14:26:42.3739105Z rootdir: /var/lib/jenkins/pytorch
2025-12-04T14:26:42.3739146Z configfile: pytest.ini
2025-12-04T14:26:42.3739313Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0
2025-12-04T14:26:42.3739389Z collecting ... collected 60 items / 20 deselected / 40 selected
2025-12-04T14:26:42.3739449Z stepcurrent: skipping 20 already run items.
2025-12-04T14:26:42.3739495Z Running 0 items in this shard
2025-12-04T14:26:42.3739497Z 
2025-12-04T14:26:42.3739735Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-fb3f972c7effba9d.xml -
2025-12-04T14:26:42.3739803Z ============================ 20 deselected in 0.01s ============================
2025-12-04T14:26:42.3743228Z The following tests failed consistently: ['test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda', 'test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda']
2025-12-04T14:26:42.3743265Z 
2025-12-04T14:26:42.3743454Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_core 2/3 (test/test-reports/distributed.fsdp.test_fsdp_core_2.3_d5fedc741f801f96_.log)
2025-12-04T14:26:42.3743456Z 
2025-12-04T14:26:42.3743578Z Finished distributed/fsdp/test_fsdp_core 2/3 ... [2025-12-04 14:26:42.124764][2242226.58208604], took 23.12min
2025-12-04T14:26:42.3743855Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:26:42.3743947Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:26:42.3744042Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-12-04T14:26:42.3744094Z Uploading artifacts took 0.00 seconds
2025-12-04T14:26:42.3744148Z distributed/fsdp/test_fsdp_core 2/3 failed!
2025-12-04T14:26:42.3744282Z Running distributed/fsdp/test_distributed_checkpoint 1/1 ... [2025-12-04 14:26:42.127342][2242226.584667217]
2025-12-04T14:26:42.3744332Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:26:42.3744676Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:26:42.127511]
2025-12-04T14:26:44.4950754Z 
2025-12-04T14:26:44.4952104Z distributed/fsdp/test_distributed_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_9e87c5baebb388d8_.log
2025-12-04T14:26:44.4955088Z Running 2 items in this shard: test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type0_cuda, test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type1_cuda
2025-12-04T14:26:44.4956514Z 
2025-12-04T14:26:44.4957123Z Finished distributed/fsdp/test_distributed_checkpoint 1/1 ... [2025-12-04 14:26:44.494663][2242228.951983964], took 0.04min
2025-12-04T14:26:44.4971982Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:26:44.4978475Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:26:44.4980471Z Running distributed/test_c10d_spawn_nccl 1/1 ... [2025-12-04 14:26:44.497963][2242228.95528761]
2025-12-04T14:26:44.4980776Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:26:44.4982329Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:26:44.498129]
2025-12-04T14:27:49.4200329Z 
2025-12-04T14:27:49.4201264Z distributed/test_c10d_spawn_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_nccl_1.1_e7d1514cf8b87098_.log
2025-12-04T14:27:49.4204539Z Running 10 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_reduce_non_contiguous, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous
2025-12-04T14:27:49.4207770Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather
2025-12-04T14:27:49.4208445Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base
2025-12-04T14:27:49.4209144Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_reduce_non_contiguous
2025-12-04T14:27:49.4209831Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all
2025-12-04T14:27:49.4210493Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single
2025-12-04T14:27:49.4211151Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce
2025-12-04T14:27:49.4211779Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast
2025-12-04T14:27:49.4212294Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce
2025-12-04T14:27:49.4212782Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter
2025-12-04T14:27:49.4213307Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous
2025-12-04T14:27:49.4213614Z 
2025-12-04T14:27:49.4214322Z Finished distributed/test_c10d_spawn_nccl 1/1 ... [2025-12-04 14:27:49.419719][2242293.87703948], took 1.08min
2025-12-04T14:27:49.4222073Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:27:49.4226828Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:27:49.4229347Z Running distributed/test_c10d_gloo 2/2 ... [2025-12-04 14:27:49.422839][2242293.880164009]
2025-12-04T14:27:49.4229661Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:27:49.4231184Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=2', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:27:49.423010]
2025-12-04T14:37:14.1816882Z 
2025-12-04T14:37:14.1818047Z distributed/test_c10d_gloo 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_2.2_074ed204d1d4582c_.log
2025-12-04T14:37:14.1843426Z Running 119 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_inference_mode, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_op_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_block_current_stream_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_complex, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_scatter, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_send_recv_complex, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_long, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_short_pickle, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check
2025-12-04T14:37:14.1859028Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init
2025-12-04T14:37:14.1859355Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics
2025-12-04T14:37:14.1859662Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda
2025-12-04T14:37:14.1859990Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks
2025-12-04T14:37:14.1860299Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_inference_mode
2025-12-04T14:37:14.1860663Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input
2025-12-04T14:37:14.1860993Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async
2025-12-04T14:37:14.1861317Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks
2025-12-04T14:37:14.1861640Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress
2025-12-04T14:37:14.1861954Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_op_timeout
2025-12-04T14:37:14.1862267Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda
2025-12-04T14:37:14.1862609Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_block_current_stream_cuda
2025-12-04T14:37:14.1862920Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks
2025-12-04T14:37:14.1863218Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics
2025-12-04T14:37:14.1863514Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda
2025-12-04T14:37:14.1863811Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks
2025-12-04T14:37:14.1864118Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input
2025-12-04T14:37:14.1864432Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress
2025-12-04T14:37:14.1864725Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda
2025-12-04T14:37:14.1865024Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks
2025-12-04T14:37:14.1865314Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter
2025-12-04T14:37:14.1865652Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor
2025-12-04T14:37:14.1874953Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced
2025-12-04T14:37:14.1875294Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks
2025-12-04T14:37:14.1875600Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda
2025-12-04T14:37:14.1875911Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_complex
2025-12-04T14:37:14.1876224Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks
2025-12-04T14:37:14.1876558Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched
2025-12-04T14:37:14.1876968Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param
2025-12-04T14:37:14.1877428Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False
2025-12-04T14:37:14.1877843Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing
2025-12-04T14:37:14.1878292Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False
2025-12-04T14:37:14.1878735Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False
2025-12-04T14:37:14.1879149Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once
2025-12-04T14:37:14.1879499Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_complex_params
2025-12-04T14:37:14.1879844Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init
2025-12-04T14:37:14.1880206Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type
2025-12-04T14:37:14.1880604Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty
2025-12-04T14:37:14.1881023Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph
2025-12-04T14:37:14.1881437Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list
2025-12-04T14:37:14.1881858Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list
2025-12-04T14:37:14.1882249Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module
2025-12-04T14:37:14.1882595Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module
2025-12-04T14:37:14.1882962Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters
2025-12-04T14:37:14.1883325Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state
2025-12-04T14:37:14.1883667Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint
2025-12-04T14:37:14.1884019Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input
2025-12-04T14:37:14.1884357Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer
2025-12-04T14:37:14.1884666Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters
2025-12-04T14:37:14.1884974Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket
2025-12-04T14:37:14.1885282Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_basics
2025-12-04T14:37:14.1885617Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_basics_cuda
2025-12-04T14:37:14.1885966Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_coalesced_async
2025-12-04T14:37:14.1886325Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_noncontiguous_input
2025-12-04T14:37:14.1886694Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_stress_cuda
2025-12-04T14:37:14.1887066Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_basics
2025-12-04T14:37:14.1887423Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_basics
2025-12-04T14:37:14.1887770Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_barrier_implies_wait
2025-12-04T14:37:14.1888138Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_basics_cuda
2025-12-04T14:37:14.1888468Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_checks
2025-12-04T14:37:14.1888795Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_basics_cuda
2025-12-04T14:37:14.1889121Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_stress
2025-12-04T14:37:14.1889459Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_multi_device_constructor
2025-12-04T14:37:14.1889794Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_basics
2025-12-04T14:37:14.1890118Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_basics_cuda
2025-12-04T14:37:14.1890444Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_checks
2025-12-04T14:37:14.1890764Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_scatter
2025-12-04T14:37:14.1891087Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_stress_cuda
2025-12-04T14:37:14.1891419Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_basics_cuda
2025-12-04T14:37:14.1891744Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_stress
2025-12-04T14:37:14.1892070Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_stress_cuda
2025-12-04T14:37:14.1892400Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_send_recv_complex
2025-12-04T14:37:14.1892749Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_basics_cuda
2025-12-04T14:37:14.1893103Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_checks
2025-12-04T14:37:14.1893465Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_cuda_dispatched
2025-12-04T14:37:14.1893809Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_checks
2025-12-04T14:37:14.1894137Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_into_tensor_coalesced
2025-12-04T14:37:14.1894487Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_noncontiguous_input
2025-12-04T14:37:14.1894817Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_basics
2025-12-04T14:37:14.1895126Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_checks
2025-12-04T14:37:14.1895443Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_async
2025-12-04T14:37:14.1895774Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_basics
2025-12-04T14:37:14.1896132Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_checks
2025-12-04T14:37:14.1896483Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_checks_cuda
2025-12-04T14:37:14.1896862Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_stress_cuda
2025-12-04T14:37:14.1897198Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_barrier_implies_wait
2025-12-04T14:37:14.1897524Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_basics
2025-12-04T14:37:14.1897833Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_basics_cuda
2025-12-04T14:37:14.1898147Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_checks
2025-12-04T14:37:14.1898449Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_empty_tensors
2025-12-04T14:37:14.1898750Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_basics
2025-12-04T14:37:14.1899065Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_noncontiguous_input
2025-12-04T14:37:14.1899379Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_stress
2025-12-04T14:37:14.1899672Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_long
2025-12-04T14:37:14.1899980Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_multi_device_constructor
2025-12-04T14:37:14.1900314Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_basics
2025-12-04T14:37:14.1900620Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_basics_cuda
2025-12-04T14:37:14.1900935Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_scatter_tensor
2025-12-04T14:37:14.1901243Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_stress
2025-12-04T14:37:14.1901546Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_stress_cuda
2025-12-04T14:37:14.1901852Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_basics
2025-12-04T14:37:14.1902152Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_checks
2025-12-04T14:37:14.1902450Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_stress
2025-12-04T14:37:14.1902748Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_short_pickle
2025-12-04T14:37:14.1903060Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_basics
2025-12-04T14:37:14.1903400Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_cuda_dispatched
2025-12-04T14:37:14.1903711Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors
2025-12-04T14:37:14.1903988Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group
2025-12-04T14:37:14.1904290Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default
2025-12-04T14:37:14.1904609Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup
2025-12-04T14:37:14.1904951Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch
2025-12-04T14:37:14.1905297Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single
2025-12-04T14:37:14.1905716Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced
2025-12-04T14:37:14.1906115Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives
2025-12-04T14:37:14.1906541Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_default_process_group
2025-12-04T14:37:14.1907023Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2025-12-04T14:37:14.1907457Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier
2025-12-04T14:37:14.1907807Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync
2025-12-04T14:37:14.1908114Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check
2025-12-04T14:37:14.1908294Z 
2025-12-04T14:37:14.1908416Z Finished distributed/test_c10d_gloo 2/2 ... [2025-12-04 14:37:14.182581][2242858.639902119], took 9.41min
2025-12-04T14:37:14.1908853Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:37:14.1909253Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:37:14.1909481Z Running distributed/test_launcher 1/1 ... [2025-12-04 14:37:14.185338][2242858.642662478]
2025-12-04T14:37:14.1909674Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:37:14.1910072Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_launcher.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:37:14.185508]
2025-12-04T14:37:17.2552437Z 
2025-12-04T14:37:17.2553466Z distributed/test_launcher 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_launcher_1.1_de13b735cc62c3d7_.log
2025-12-04T14:37:17.2554553Z Running 1 items in this shard: test/distributed/test_launcher.py::TestDistributedLaunch::test_launch_user_script
2025-12-04T14:37:17.2554982Z 
2025-12-04T14:37:17.2555270Z Finished distributed/test_launcher 1/1 ... [2025-12-04 14:37:17.254875][2242861.712194322], took 0.05min
2025-12-04T14:37:17.2578151Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:37:17.2583978Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:37:17.2585640Z Running distributed/test_store 1/1 ... [2025-12-04 14:37:17.258477][2242861.715801239]
2025-12-04T14:37:17.2585978Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:37:17.2587789Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_store.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:37:17.258650]
2025-12-04T14:42:03.8209424Z 
2025-12-04T14:42:03.8210431Z distributed/test_store 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_store_1.1_cc68795341bf8a46_.log
2025-12-04T14:42:03.8231978Z Running 126 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append, test/distributed/test_store.py::FileStoreTest::test_clone, test/distributed/test_store.py::FileStoreTest::test_compare_set, test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file, test/distributed/test_store.py::FileStoreTest::test_list_keys, test/distributed/test_store.py::FileStoreTest::test_multi_get, test/distributed/test_store.py::FileStoreTest::test_multi_set, test/distributed/test_store.py::FileStoreTest::test_queues, test/distributed/test_store.py::FileStoreTest::test_queues_bidirectional, test/distributed/test_store.py::FileStoreTest::test_queues_nonblocking, test/distributed/test_store.py::FileStoreTest::test_queues_timeout, test/distributed/test_store.py::FileStoreTest::test_refcount, test/distributed/test_store.py::FileStoreTest::test_set_get_check, test/distributed/test_store.py::FileStoreTest::test_simple_wait, test/distributed/test_store.py::HashStoreTest::test_append, test/distributed/test_store.py::HashStoreTest::test_clone, test/distributed/test_store.py::HashStoreTest::test_compare_set, test/distributed/test_store.py::HashStoreTest::test_list_keys, test/distributed/test_store.py::HashStoreTest::test_multi_get, test/distributed/test_store.py::HashStoreTest::test_multi_set, test/distributed/test_store.py::HashStoreTest::test_queues, test/distributed/test_store.py::HashStoreTest::test_queues_bidirectional, test/distributed/test_store.py::HashStoreTest::test_queues_nonblocking, test/distributed/test_store.py::HashStoreTest::test_queues_timeout, test/distributed/test_store.py::HashStoreTest::test_set_get_check, test/distributed/test_store.py::HashStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store, test/distributed/test_store.py::PrefixFileStoreTest::test_append, test/distributed/test_store.py::PrefixFileStoreTest::test_clone, test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set, test/distributed/test_store.py::PrefixFileStoreTest::test_list_keys, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set, test/distributed/test_store.py::PrefixFileStoreTest::test_queues, test/distributed/test_store.py::PrefixFileStoreTest::test_queues_bidirectional, test/distributed/test_store.py::PrefixFileStoreTest::test_queues_nonblocking, test/distributed/test_store.py::PrefixFileStoreTest::test_queues_timeout, test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::TCPStoreTest::test_agent_store, test/distributed/test_store.py::TCPStoreTest::test_append, test/distributed/test_store.py::TCPStoreTest::test_clone, test/distributed/test_store.py::TCPStoreTest::test_compare_set, test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::TCPStoreTest::test_list_keys, test/distributed/test_store.py::TCPStoreTest::test_multi_get, test/distributed/test_store.py::TCPStoreTest::test_multi_set, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multitenancy, test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::TCPStoreTest::test_queues, test/distributed/test_store.py::TCPStoreTest::test_queues_bidirectional, test/distributed/test_store.py::TCPStoreTest::test_queues_nonblocking, test/distributed/test_store.py::TCPStoreTest::test_queues_timeout, test/distributed/test_store.py::TCPStoreTest::test_repr, test/distributed/test_store.py::TCPStoreTest::test_set_get_check, test/distributed/test_store.py::TCPStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::LibUvTCPStoreTest::test_agent_store, test/distributed/test_store.py::LibUvTCPStoreTest::test_append, test/distributed/test_store.py::LibUvTCPStoreTest::test_clone, test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_list_keys, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy, test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::LibUvTCPStoreTest::test_queues, test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_bidirectional, test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_nonblocking, test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_timeout, test/distributed/test_store.py::LibUvTCPStoreTest::test_repr, test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check, test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait, test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::PrefixTCPStoreTest::test_append, test/distributed/test_store.py::PrefixTCPStoreTest::test_clone, test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_list_keys, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_queues, test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_bidirectional, test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_nonblocking, test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_timeout, test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store, test/distributed/test_store.py::PythonStoreTest::test_set_get, test/distributed/test_store.py::RendezvousTest::test_unknown_handler, test/distributed/test_store.py::RendezvousTest::test_url_with_node_params, test/distributed/test_store.py::RendezvousEnvTest::test_nominal, test/distributed/test_store.py::RendezvousFileTest::test_common_errors, test/distributed/test_store.py::RendezvousFileTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_common_errors, test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout, test/distributed/test_store.py::RendezvousTCPTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv, test/distributed/test_store.py::TestPythonStore::test_append_roundtrip, test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip, test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv, test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param, test/distributed/test_store.py::TestClientProtocol::test_client_connect
2025-12-04T14:42:03.8245527Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append
2025-12-04T14:42:03.8245778Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_clone
2025-12-04T14:42:03.8246038Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_compare_set
2025-12-04T14:42:03.8246329Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file
2025-12-04T14:42:03.8246614Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_list_keys
2025-12-04T14:42:03.8246908Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_get
2025-12-04T14:42:03.8247165Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_set
2025-12-04T14:42:03.8247422Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_queues
2025-12-04T14:42:03.8247689Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_queues_bidirectional
2025-12-04T14:42:03.8247978Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_queues_nonblocking
2025-12-04T14:42:03.8248258Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_queues_timeout
2025-12-04T14:42:03.8248554Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_refcount
2025-12-04T14:42:03.8248823Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_set_get_check
2025-12-04T14:42:03.8249098Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_simple_wait
2025-12-04T14:42:03.8249354Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_append
2025-12-04T14:42:03.8249602Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_clone
2025-12-04T14:42:03.8249858Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_compare_set
2025-12-04T14:42:03.8250116Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_list_keys
2025-12-04T14:42:03.8250370Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_get
2025-12-04T14:42:03.8250627Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_set
2025-12-04T14:42:03.8250877Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_queues
2025-12-04T14:42:03.8251146Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_queues_bidirectional
2025-12-04T14:42:03.8251432Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_queues_nonblocking
2025-12-04T14:42:03.8251707Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_queues_timeout
2025-12-04T14:42:03.8251979Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_set_get_check
2025-12-04T14:42:03.8252243Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_simple_wait
2025-12-04T14:42:03.8252522Z Running 1 items in this shard: test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store
2025-12-04T14:42:03.8252840Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_append
2025-12-04T14:42:03.8253105Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_clone
2025-12-04T14:42:03.8253398Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set
2025-12-04T14:42:03.8253678Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_list_keys
2025-12-04T14:42:03.8253953Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get
2025-12-04T14:42:03.8254283Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set
2025-12-04T14:42:03.8254555Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_queues
2025-12-04T14:42:03.8254840Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_queues_bidirectional
2025-12-04T14:42:03.8255141Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_queues_nonblocking
2025-12-04T14:42:03.8255433Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_queues_timeout
2025-12-04T14:42:03.8255717Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check
2025-12-04T14:42:03.8255998Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait
2025-12-04T14:42:03.8256283Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use
2025-12-04T14:42:03.8256558Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_agent_store
2025-12-04T14:42:03.8256836Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_append
2025-12-04T14:42:03.8257080Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_clone
2025-12-04T14:42:03.8257337Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_compare_set
2025-12-04T14:42:03.8257627Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket
2025-12-04T14:42:03.8257912Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_list_keys
2025-12-04T14:42:03.8258165Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_get
2025-12-04T14:42:03.8258417Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_set
2025-12-04T14:42:03.8258705Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size
2025-12-04T14:42:03.8259026Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size
2025-12-04T14:42:03.8259319Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multitenancy
2025-12-04T14:42:03.8259587Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys
2025-12-04T14:42:03.8259844Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_queues
2025-12-04T14:42:03.8260110Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_queues_bidirectional
2025-12-04T14:42:03.8260390Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_queues_nonblocking
2025-12-04T14:42:03.8260667Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_queues_timeout
2025-12-04T14:42:03.8260919Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_repr
2025-12-04T14:42:03.8261172Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_set_get_check
2025-12-04T14:42:03.8261434Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_simple_wait
2025-12-04T14:42:03.8261740Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients
2025-12-04T14:42:03.8262040Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket
2025-12-04T14:42:03.8262343Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises
2025-12-04T14:42:03.8262635Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use
2025-12-04T14:42:03.8262938Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_agent_store
2025-12-04T14:42:03.8263231Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_append
2025-12-04T14:42:03.8263488Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_clone
2025-12-04T14:42:03.8263754Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set
2025-12-04T14:42:03.8264059Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket
2025-12-04T14:42:03.8264354Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_list_keys
2025-12-04T14:42:03.8264620Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get
2025-12-04T14:42:03.8264885Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set
2025-12-04T14:42:03.8265186Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size
2025-12-04T14:42:03.8265522Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size
2025-12-04T14:42:03.8265830Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy
2025-12-04T14:42:03.8266113Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys
2025-12-04T14:42:03.8266383Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_queues
2025-12-04T14:42:03.8266666Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_bidirectional
2025-12-04T14:42:03.8266989Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_nonblocking
2025-12-04T14:42:03.8267274Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_timeout
2025-12-04T14:42:03.8267546Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_repr
2025-12-04T14:42:03.8267813Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check
2025-12-04T14:42:03.8268090Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait
2025-12-04T14:42:03.8268394Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients
2025-12-04T14:42:03.8268708Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket
2025-12-04T14:42:03.8269008Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises
2025-12-04T14:42:03.8269286Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_append
2025-12-04T14:42:03.8269551Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_clone
2025-12-04T14:42:03.8269823Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set
2025-12-04T14:42:03.8270100Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_list_keys
2025-12-04T14:42:03.8270373Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get
2025-12-04T14:42:03.8270667Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set
2025-12-04T14:42:03.8270935Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_queues
2025-12-04T14:42:03.8271238Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_bidirectional
2025-12-04T14:42:03.8271534Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_nonblocking
2025-12-04T14:42:03.8271845Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_timeout
2025-12-04T14:42:03.8272152Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check
2025-12-04T14:42:03.8272432Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait
2025-12-04T14:42:03.8272732Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store
2025-12-04T14:42:03.8273026Z Running 1 items in this shard: test/distributed/test_store.py::PythonStoreTest::test_set_get
2025-12-04T14:42:03.8273293Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_unknown_handler
2025-12-04T14:42:03.8273576Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_url_with_node_params
2025-12-04T14:42:03.8273851Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousEnvTest::test_nominal
2025-12-04T14:42:03.8274128Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_common_errors
2025-12-04T14:42:03.8274402Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_nominal
2025-12-04T14:42:03.8274672Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_common_errors
2025-12-04T14:42:03.8274949Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout
2025-12-04T14:42:03.8275217Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_nominal
2025-12-04T14:42:03.8275516Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client
2025-12-04T14:42:03.8275838Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set
2025-12-04T14:42:03.8276142Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv
2025-12-04T14:42:03.8276436Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_append_roundtrip
2025-12-04T14:42:03.8276728Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks
2025-12-04T14:42:03.8277068Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough
2025-12-04T14:42:03.8277373Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip
2025-12-04T14:42:03.8277666Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip
2025-12-04T14:42:03.8277952Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip
2025-12-04T14:42:03.8278242Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail
2025-12-04T14:42:03.8278535Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store
2025-12-04T14:42:03.8278825Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store
2025-12-04T14:42:03.8279126Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store
2025-12-04T14:42:03.8279427Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store
2025-12-04T14:42:03.8279749Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv
2025-12-04T14:42:03.8280064Z Running 1 items in this shard: test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait
2025-12-04T14:42:03.8280354Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var
2025-12-04T14:42:03.8280642Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param
2025-12-04T14:42:03.8280960Z Running 1 items in this shard: test/distributed/test_store.py::TestClientProtocol::test_client_connect
2025-12-04T14:42:03.8281123Z 
2025-12-04T14:42:03.8281238Z Finished distributed/test_store 1/1 ... [2025-12-04 14:42:03.821713][2243148.27903355], took 4.78min
2025-12-04T14:42:03.8281670Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:42:03.8282070Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:42:03.8282299Z Running distributed/test_c10d_nccl 1/3 ... [2025-12-04 14:42:03.824690][2243148.282014394]
2025-12-04T14:42:03.8282491Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:42:03.8282915Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:42:03.824855]
2025-12-04T14:52:58.7731592Z 
2025-12-04T14:52:58.7732667Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_76f56658f2038bd5_.log
2025-12-04T14:52:58.7758167Z Running 91 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_block_current_stream, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_with_idx, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_backend_properties, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_multiple_comms, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_multiple_exclusions, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_validation, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_mixed_real_and_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_error_detection_and_propagation, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_restart_pg_after_error, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_multiple_resets_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_circular_buffer_full_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_partial_overwrite_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_partial_overwrite_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale
2025-12-04T14:52:58.7772924Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors
2025-12-04T14:52:58.7773230Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus
2025-12-04T14:52:58.7773560Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs
2025-12-04T14:52:58.7773882Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg
2025-12-04T14:52:58.7774198Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_block_current_stream
2025-12-04T14:52:58.7774532Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False
2025-12-04T14:52:58.7774875Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True
2025-12-04T14:52:58.7775202Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group
2025-12-04T14:52:58.7775546Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl
2025-12-04T14:52:58.7775934Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check
2025-12-04T14:52:58.7776272Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout
2025-12-04T14:52:58.7776624Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_with_idx
2025-12-04T14:52:58.7776984Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16
2025-12-04T14:52:58.7777344Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2
2025-12-04T14:52:58.7777677Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True
2025-12-04T14:52:58.7778005Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init
2025-12-04T14:52:58.7778348Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_backend_properties
2025-12-04T14:52:58.7778704Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_multiple_comms
2025-12-04T14:52:58.7779057Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_multiple_exclusions
2025-12-04T14:52:58.7779405Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_validation
2025-12-04T14:52:58.7779749Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module
2025-12-04T14:52:58.7780103Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view
2025-12-04T14:52:58.7780442Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output
2025-12-04T14:52:58.7780797Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing
2025-12-04T14:52:58.7781193Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False
2025-12-04T14:52:58.7781603Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False
2025-12-04T14:52:58.7782011Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl
2025-12-04T14:52:58.7782371Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params
2025-12-04T14:52:58.7782721Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_mixed_real_and_complex_params
2025-12-04T14:52:58.7783076Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters
2025-12-04T14:52:58.7783436Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view
2025-12-04T14:52:58.7783837Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info
2025-12-04T14:52:58.7784221Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off
2025-12-04T14:52:58.7784626Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail
2025-12-04T14:52:58.7785064Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off
2025-12-04T14:52:58.7785419Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16
2025-12-04T14:52:58.7785760Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl
2025-12-04T14:52:58.7786093Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view
2025-12-04T14:52:58.7786505Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess
2025-12-04T14:52:58.7790911Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module
2025-12-04T14:52:58.7791257Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module
2025-12-04T14:52:58.7791644Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None
2025-12-04T14:52:58.7792061Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids
2025-12-04T14:52:58.7792449Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason
2025-12-04T14:52:58.7792780Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad
2025-12-04T14:52:58.7793107Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error
2025-12-04T14:52:58.7793491Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl
2025-12-04T14:52:58.7793831Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object
2025-12-04T14:52:58.7794144Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast
2025-12-04T14:52:58.7794443Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq
2025-12-04T14:52:58.7794741Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp
2025-12-04T14:52:58.7795069Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_error_detection_and_propagation
2025-12-04T14:52:58.7795413Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env
2025-12-04T14:52:58.7795761Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier
2025-12-04T14:52:58.7796098Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_restart_pg_after_error
2025-12-04T14:52:58.7796401Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config
2025-12-04T14:52:58.7796695Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced
2025-12-04T14:52:58.7797063Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default
2025-12-04T14:52:58.7797382Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup
2025-12-04T14:52:58.7797691Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group
2025-12-04T14:52:58.7797973Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor
2025-12-04T14:52:58.7798326Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base
2025-12-04T14:52:58.7798742Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2
2025-12-04T14:52:58.7799192Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2025-12-04T14:52:58.7799611Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True
2025-12-04T14:52:58.7799997Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False
2025-12-04T14:52:58.7800446Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True
2025-12-04T14:52:58.7800801Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True
2025-12-04T14:52:58.7801120Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False
2025-12-04T14:52:58.7801456Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False
2025-12-04T14:52:58.7801790Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True
2025-12-04T14:52:58.7802145Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False
2025-12-04T14:52:58.7802534Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False
2025-12-04T14:52:58.7802914Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False
2025-12-04T14:52:58.7803281Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False
2025-12-04T14:52:58.7803616Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata
2025-12-04T14:52:58.7803935Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_False
2025-12-04T14:52:58.7804236Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe
2025-12-04T14:52:58.7804547Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_multiple_resets_timing_enabled_False
2025-12-04T14:52:58.7804916Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_circular_buffer_full_timing_enabled_False
2025-12-04T14:52:58.7805295Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_partial_overwrite_timing_enabled_False
2025-12-04T14:52:58.7805667Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_partial_overwrite_timing_enabled_True
2025-12-04T14:52:58.7806036Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True
2025-12-04T14:52:58.7806346Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long
2025-12-04T14:52:58.7806671Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False
2025-12-04T14:52:58.7807104Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False
2025-12-04T14:52:58.7807517Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True
2025-12-04T14:52:58.7807902Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False
2025-12-04T14:52:58.7808285Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True
2025-12-04T14:52:58.7808632Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired
2025-12-04T14:52:58.7809009Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale
2025-12-04T14:52:58.7809239Z 
2025-12-04T14:52:58.7809360Z Finished distributed/test_c10d_nccl 1/3 ... [2025-12-04 14:52:58.773802][2243803.231122447], took 10.92min
2025-12-04T14:52:58.7809794Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:52:58.7810193Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:52:58.7810414Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-12-04T14:52:58.7810597Z Uploading artifacts took 0.00 seconds
2025-12-04T14:52:58.7810786Z Running distributed/test_c10d_ops_nccl 1/1 ... [2025-12-04 14:52:58.776923][2243803.234247428]
2025-12-04T14:52:58.7810977Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:52:58.7811380Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/test_c10d_ops_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:52:58.777086]
2025-12-04T14:53:07.6044169Z 
2025-12-04T14:53:07.6045294Z distributed/test_c10d_ops_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ops_nccl_1.1_2fce8ff794ce61fe_.log
2025-12-04T14:53:07.6056622Z Running 30 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_all_gather_v, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_float8, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_bfloat16, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_float8, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_v, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook
2025-12-04T14:53:07.6064721Z 
2025-12-04T14:53:07.6064975Z Finished distributed/test_c10d_ops_nccl 1/1 ... [2025-12-04 14:53:07.604014][2243812.061334966], took 0.15min
2025-12-04T14:53:07.6069353Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:07.6075073Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:53:07.6077635Z Running distributed/elastic/events/lib_test 1/1 ... [2025-12-04 14:53:07.607639][2243812.064963998]
2025-12-04T14:53:07.6077917Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:53:07.6079436Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/events/lib_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:53:07.607819]
2025-12-04T14:53:09.6251845Z 
2025-12-04T14:53:09.6253057Z distributed/elastic/events/lib_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.events.lib_test_1.1_bac486344c9deb76_.log
2025-12-04T14:53:09.6254523Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str
2025-12-04T14:53:09.6255670Z 
2025-12-04T14:53:09.6255811Z Finished distributed/elastic/events/lib_test 1/1 ... [2025-12-04 14:53:09.624881][2243814.082201901], took 0.03min
2025-12-04T14:53:09.6276031Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:09.6282254Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:53:09.6285912Z Running distributed/elastic/metrics/api_test 1/1 ... [2025-12-04 14:53:09.628364][2243814.085688526]
2025-12-04T14:53:09.6286470Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:53:09.6287618Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:53:09.628533]
2025-12-04T14:53:11.7461463Z 
2025-12-04T14:53:11.7462590Z distributed/elastic/metrics/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.metrics.api_test_1.1_36ff046d0d90b58f_.log
2025-12-04T14:53:11.7464921Z Running 3 items in this shard: test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_get_metric_name, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_inheritance, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_profile
2025-12-04T14:53:11.7465976Z 
2025-12-04T14:53:11.7466336Z Finished distributed/elastic/metrics/api_test 1/1 ... [2025-12-04 14:53:11.745796][2243816.203116755], took 0.04min
2025-12-04T14:53:11.7488319Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:11.7494408Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:53:11.7495949Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2025-12-04 14:53:11.749504][2243816.206828606]
2025-12-04T14:53:11.7496390Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:53:11.7498123Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:53:11.749671]
2025-12-04T14:53:32.2467271Z 
2025-12-04T14:53:32.2469050Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_50d9a5a39de8a0ed_.log
2025-12-04T14:53:32.2481318Z Running 26 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait_on_a_child_thread, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_wait_for_all_child_procs_to_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsFuncTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary_duplicate_log_filters, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary_redirect_and_tee
2025-12-04T14:53:32.2488582Z 
2025-12-04T14:53:32.2488865Z Finished distributed/elastic/multiprocessing/api_test 1/1 ... [2025-12-04 14:53:32.246404][2243836.703724504], took 0.34min
2025-12-04T14:53:32.2493899Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:32.2499499Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:53:32.2501362Z Running distributed/elastic/timer/local_timer_example 1/1 ... [2025-12-04 14:53:32.250034][2243836.707358786]
2025-12-04T14:53:32.2501628Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:53:32.2503120Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:53:32.250201]
2025-12-04T14:53:42.4316106Z 
2025-12-04T14:53:42.4317887Z distributed/elastic/timer/local_timer_example 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_example_1.1_e42fea71615785ac_.log
2025-12-04T14:53:42.4319519Z Running 2 items in this shard: test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_example_start_method_spawn, test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_torch_mp_example
2025-12-04T14:53:42.4320320Z 
2025-12-04T14:53:42.4320657Z Finished distributed/elastic/timer/local_timer_example 1/1 ... [2025-12-04 14:53:42.431100][2243846.888419134], took 0.17min
2025-12-04T14:53:42.4340971Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:42.4347114Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:53:42.4349242Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2025-12-04 14:53:42.434816][2243846.892139735]
2025-12-04T14:53:42.4349624Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:53:42.4351462Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:53:42.435020]
2025-12-04T14:53:48.3089235Z 
2025-12-04T14:53:48.3090082Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_e0d961f21b2d7da0_.log
2025-12-04T14:53:48.3092808Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue
2025-12-04T14:53:48.3094892Z 
2025-12-04T14:53:48.3095041Z Finished distributed/elastic/timer/local_timer_test 1/1 ... [2025-12-04 14:53:48.308630][2243852.765949453], took 0.10min
2025-12-04T14:53:48.3109278Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:48.3115297Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:53:48.3117523Z Running distributed/elastic/utils/distributed_test 1/1 ... [2025-12-04 14:53:48.311661][2243852.768985624]
2025-12-04T14:53:48.3117758Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:53:48.3119369Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:53:48.311838]
2025-12-04T14:53:54.0843762Z 
2025-12-04T14:53:54.0844584Z distributed/elastic/utils/distributed_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.distributed_test_1.1_dfdef03d38281e35_.log
2025-12-04T14:53:54.0846232Z Running 8 items in this shard: test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_no_port_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_single_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_worker, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_with_libuv_support, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_worker
2025-12-04T14:53:54.0847668Z 
2025-12-04T14:53:54.0847837Z Finished distributed/elastic/utils/distributed_test 1/1 ... [2025-12-04 14:53:54.084049][2243858.541371394], took 0.10min
2025-12-04T14:53:54.0863104Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:54.0869499Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:53:54.0871814Z Running distributed/elastic/utils/logging_test 1/1 ... [2025-12-04 14:53:54.087099][2243858.544422675]
2025-12-04T14:53:54.0872042Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:53:54.0873842Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:53:54.087289]
2025-12-04T14:53:56.1551743Z 
2025-12-04T14:53:56.1552920Z distributed/elastic/utils/logging_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.logging_test_1.1_8ba8bc2f8dd7cff0_.log
2025-12-04T14:53:56.1554214Z Running 2 items in this shard: test/distributed/elastic/utils/logging_test.py::LoggingTest::test_derive_module_name, test/distributed/elastic/utils/logging_test.py::LoggingTest::test_logger_name
2025-12-04T14:53:56.1554866Z 
2025-12-04T14:53:56.1555778Z Finished distributed/elastic/utils/logging_test 1/1 ... [2025-12-04 14:53:56.154848][2243860.61216786], took 0.03min
2025-12-04T14:53:56.1581174Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:56.1586883Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:53:56.1588958Z Running distributed/elastic/utils/util_test 1/1 ... [2025-12-04 14:53:56.158780][2243860.616105257]
2025-12-04T14:53:56.1589585Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-12-04T14:53:56.1590684Z Executing ['/opt/conda/envs/py_3.12/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 14:53:56.158956]
2025-12-04T14:53:58.3769904Z 
2025-12-04T14:53:58.3771111Z distributed/elastic/utils/util_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.util_test_1.1_62b5ecfdbc14b24e_.log
2025-12-04T14:53:58.3774952Z Running 12 items in this shard: test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_hash_store, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_operations, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_rank_tracing, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_0, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_n, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize_hash_store, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_custom_name, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_different, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_none
2025-12-04T14:53:58.3778383Z 
2025-12-04T14:53:58.3778698Z Finished distributed/elastic/utils/util_test 1/1 ... [2025-12-04 14:53:58.376644][2243862.833964082], took 0.04min
2025-12-04T14:53:58.3796447Z Parsing testcases for test report: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-96780e4a4f50efe4.xml
2025-12-04T14:53:58.3802778Z Failed to parse and upload json test reports: Unable to locate credentials
2025-12-04T14:54:00.4154020Z Running test batch 'tests to run' cost 8735.73 seconds
2025-12-04T14:54:00.4156876Z Emitting td_test_failure_stats_v2
2025-12-04T14:54:00.4160776Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764860040_1160aafed12111f0a2d44200ea89c3d6
2025-12-04T14:54:02.4343237Z /var/lib/jenkins/pytorch/tools/stats/upload_metrics.py:156: UserWarning: Error uploading metric td_test_failure_stats_v2 to DynamoDB: Unable to locate credentials
2025-12-04T14:54:02.4344265Z   warn(f"Error uploading metric {metric_name} to DynamoDB: {e}")
2025-12-04T14:54:02.4344734Z Emitting td_test_failure_stats_v2
2025-12-04T14:54:02.4345451Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764860042_1294a33ad12111f0a2d44200ea89c3d6
2025-12-04T14:54:02.4361044Z Emitting td_test_failure_stats_v2
2025-12-04T14:54:02.4361617Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764860042_1294e854d12111f0a2d44200ea89c3d6
2025-12-04T14:54:02.4377185Z distributed/fsdp/test_fsdp_input 1/1 failed!
2025-12-04T14:54:02.4377514Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 failed!
2025-12-04T14:54:02.4377813Z distributed/fsdp/test_fsdp_core 2/3 failed!
2025-12-04T14:54:02.9940246Z 
2025-12-04T14:54:02.9941488Z real	145m41.130s
2025-12-04T14:54:02.9941865Z user	393m11.652s
2025-12-04T14:54:02.9943886Z sys	371m10.595s
2025-12-04T14:54:02.9944223Z + sccache_epilogue
2025-12-04T14:54:02.9944566Z + echo '::group::Sccache Compilation Log'
2025-12-04T14:54:02.9945326Z ##[group]Sccache Compilation Log
2025-12-04T14:54:02.9945725Z + echo '=================== sccache compilation log ==================='
2025-12-04T14:54:02.9946178Z =================== sccache compilation log ===================
2025-12-04T14:54:02.9947130Z + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log
2025-12-04T14:54:03.0013227Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
2025-12-04T14:54:03.0013791Z =========== If your build fails, please take a look at the log above for possible reasons ===========
2025-12-04T14:54:03.0014265Z + sccache --show-stats
2025-12-04T14:54:03.0036159Z Compile requests                    403
2025-12-04T14:54:03.0036605Z Compile requests executed             0
2025-12-04T14:54:03.0036934Z Cache hits                            0
2025-12-04T14:54:03.0037156Z Cache misses                          0
2025-12-04T14:54:03.0037427Z Cache hits rate                       -
2025-12-04T14:54:03.0037661Z Cache timeouts                        0
2025-12-04T14:54:03.0037873Z Cache read errors                     0
2025-12-04T14:54:03.0038154Z Forced recaches                       0
2025-12-04T14:54:03.0038379Z Cache write errors                    0
2025-12-04T14:54:03.0038632Z Cache errors                          0
2025-12-04T14:54:03.0038870Z Compilations                          0
2025-12-04T14:54:03.0039099Z Compilation failures                  0
2025-12-04T14:54:03.0039375Z Non-cacheable compilations            0
2025-12-04T14:54:03.0039604Z Non-cacheable calls                   7
2025-12-04T14:54:03.0039849Z Non-compilation calls               396
2025-12-04T14:54:03.0040278Z Unsupported compiler calls            0
2025-12-04T14:54:03.0040527Z Average cache write               0.000 s
2025-12-04T14:54:03.0040790Z Average compiler                  0.000 s
2025-12-04T14:54:03.0041037Z Average cache read hit            0.000 s
2025-12-04T14:54:03.0041511Z Failed distributed compilations       0
2025-12-04T14:54:03.0041677Z 
2025-12-04T14:54:03.0041794Z Non-cacheable reasons:
2025-12-04T14:54:03.0042006Z -E                                    7
2025-12-04T14:54:03.0042146Z 
2025-12-04T14:54:03.0042311Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-12-04T14:54:03.0042622Z Use direct/preprocessor mode?   yes
2025-12-04T14:54:03.0042859Z Version (client)                0.10.0
2025-12-04T14:54:03.0043111Z Max cache size                       10 GiB
2025-12-04T14:54:03.0043352Z + sccache --stop-server
2025-12-04T14:54:03.0059262Z Stopping sccache server...
2025-12-04T14:54:03.0062075Z Compile requests                    403
2025-12-04T14:54:03.0062686Z Compile requests executed             0
2025-12-04T14:54:03.0063101Z Cache hits                            0
2025-12-04T14:54:03.0063408Z Cache misses                          0
2025-12-04T14:54:03.0063700Z Cache hits rate                       -
2025-12-04T14:54:03.0064034Z Cache timeouts                        0
2025-12-04T14:54:03.0064314Z Cache read errors                     0
2025-12-04T14:54:03.0064584Z Forced recaches                       0
2025-12-04T14:54:03.0064902Z Cache write errors                    0
2025-12-04T14:54:03.0065180Z Cache errors                          0
2025-12-04T14:54:03.0065499Z Compilations                          0
2025-12-04T14:54:03.0065793Z Compilation failures                  0
2025-12-04T14:54:03.0066549Z Non-cacheable compilations            0
2025-12-04T14:54:03.0066945Z Non-cacheable calls                   7
2025-12-04T14:54:03.0067228Z Non-compilation calls               396
2025-12-04T14:54:03.0067523Z Unsupported compiler calls            0
2025-12-04T14:54:03.0068019Z Average cache write               0.000 s
2025-12-04T14:54:03.0068339Z Average compiler                  0.000 s
2025-12-04T14:54:03.0068650Z Average cache read hit            0.000 s
2025-12-04T14:54:03.0068965Z Failed distributed compilations       0
2025-12-04T14:54:03.0069158Z 
2025-12-04T14:54:03.0069282Z Non-cacheable reasons:
2025-12-04T14:54:03.0069556Z -E                                    7
2025-12-04T14:54:03.0069703Z 
2025-12-04T14:54:03.0069871Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-12-04T14:54:03.0070229Z Use direct/preprocessor mode?   yes
2025-12-04T14:54:03.0070538Z Version (client)                0.10.0
2025-12-04T14:54:03.0070793Z Max cache size                       10 GiB
2025-12-04T14:54:03.0071098Z + echo ::endgroup::
2025-12-04T14:54:03.0071622Z ##[endgroup]
2025-12-04T14:54:03.0127362Z ##[error]Process completed with exit code 1.
2025-12-04T14:54:03.0160727Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-12-04T14:54:03.0161085Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-12-04T14:54:03.0161505Z docker exec -t "4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test"
2025-12-04T14:54:03.0165828Z shell: /usr/bin/bash -e {0}
2025-12-04T14:54:03.0165961Z env:
2025-12-04T14:54:03.0166082Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:03.0166271Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:03.0166497Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:03.0166726Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:03.0167347Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:03.0167895Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:03.0168046Z   AWS_REGION: us-east-1
2025-12-04T14:54:03.0168241Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:03.0168689Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:03.0170742Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:03.0170955Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:03.0171173Z ##[endgroup]
2025-12-04T14:54:03.0889514Z ##[group]Run docker exec -t "4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca" sh -c "sudo chown -R 1001:1001 test"
2025-12-04T14:54:03.0889925Z docker exec -t "4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca" sh -c "sudo chown -R 1001:1001 test"
2025-12-04T14:54:03.0894412Z shell: /usr/bin/bash -e {0}
2025-12-04T14:54:03.0894533Z env:
2025-12-04T14:54:03.0894635Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:03.0894777Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:03.0894959Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:03.0895131Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:03.0895646Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:03.0896149Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:03.0896275Z   AWS_REGION: us-east-1
2025-12-04T14:54:03.0896447Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:03.0896608Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:03.0898812Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:03.0898987Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:03.0899174Z ##[endgroup]
2025-12-04T14:54:03.1667391Z ##[group]Run cat test/**/*_toprint.log || true
2025-12-04T14:54:03.1667563Z cat test/**/*_toprint.log || true
2025-12-04T14:54:03.1671079Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-12-04T14:54:03.1671240Z env:
2025-12-04T14:54:03.1671346Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:03.1671498Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:03.1671701Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:03.1671892Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:03.1672541Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:03.1673160Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:03.1673306Z   AWS_REGION: us-east-1
2025-12-04T14:54:03.1673456Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:03.1673623Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:03.1675854Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:03.1676044Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:03.1676247Z ##[endgroup]
2025-12-04T14:54:03.1721743Z cat: 'test/**/*_toprint.log': No such file or directory
2025-12-04T14:54:03.1786870Z Prepare all required actions
2025-12-04T14:54:03.1787300Z Getting action download info
2025-12-04T14:54:03.5322218Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-12-04T14:54:04.4320868Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-12-04T14:54:05.3422356Z ##[group]Run ./.github/actions/upload-test-artifacts
2025-12-04T14:54:05.3422517Z with:
2025-12-04T14:54:05.3422620Z   use-gha: true
2025-12-04T14:54:05.3422788Z   file-suffix: test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539
2025-12-04T14:54:05.3422977Z   s3-bucket: gha-artifacts
2025-12-04T14:54:05.3423094Z env:
2025-12-04T14:54:05.3423194Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:05.3423337Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:05.3423521Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:05.3423719Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:05.3424233Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:05.3424747Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:05.3424870Z   AWS_REGION: us-east-1
2025-12-04T14:54:05.3425032Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:05.3425191Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:05.3427247Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:05.3427426Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:05.3427616Z ##[endgroup]
2025-12-04T14:54:05.3457646Z ##[group]Run actions/upload-artifact@v4
2025-12-04T14:54:05.3457782Z with:
2025-12-04T14:54:05.3457987Z   name: test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip
2025-12-04T14:54:05.3458210Z   retention-days: 14
2025-12-04T14:54:05.3458324Z   if-no-files-found: warn
2025-12-04T14:54:05.3458442Z   path: test/**/*.json
2025-12-04T14:54:05.3458554Z   compression-level: 6
2025-12-04T14:54:05.3458664Z   overwrite: false
2025-12-04T14:54:05.3458778Z   include-hidden-files: false
2025-12-04T14:54:05.3458888Z env:
2025-12-04T14:54:05.3458982Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:05.3459120Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:05.3459300Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:05.3459467Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:05.3459974Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:05.3460460Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:05.3460579Z   AWS_REGION: us-east-1
2025-12-04T14:54:05.3460784Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:05.3460937Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:05.3462940Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:05.3463110Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:05.3463355Z ##[endgroup]
2025-12-04T14:54:05.7301218Z With the provided path, there will be 6 files uploaded
2025-12-04T14:54:05.7304607Z Artifact name is valid!
2025-12-04T14:54:05.7305173Z Root directory input is valid!
2025-12-04T14:54:05.9643667Z Beginning upload of artifact content to blob storage
2025-12-04T14:54:06.3404351Z Uploaded bytes 44615
2025-12-04T14:54:06.4046600Z Finished uploading artifact content to blob storage!
2025-12-04T14:54:06.4047803Z SHA256 digest of uploaded artifact zip is 69c4a2680aa3cd4b40bfc65623af9e094ecf3409c2bb6dd3fd3b9f16e242b170
2025-12-04T14:54:06.4049031Z Finalizing artifact upload
2025-12-04T14:54:06.5652565Z Artifact test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip.zip successfully finalized. Artifact ID 4765526920
2025-12-04T14:54:06.5653695Z Artifact test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip has been successfully uploaded! Final size is 44615 bytes. Artifact ID is 4765526920
2025-12-04T14:54:06.5657242Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/19922798714/artifacts/4765526920
2025-12-04T14:54:06.5759412Z ##[group]Run actions/upload-artifact@v4
2025-12-04T14:54:06.5759573Z with:
2025-12-04T14:54:06.5759796Z   name: test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip
2025-12-04T14:54:06.5760041Z   retention-days: 14
2025-12-04T14:54:06.5760168Z   if-no-files-found: ignore
2025-12-04T14:54:06.5760304Z   path: test/**/*.xml
test/**/*.csv

2025-12-04T14:54:06.5760445Z   compression-level: 6
2025-12-04T14:54:06.5760581Z   overwrite: false
2025-12-04T14:54:06.5760701Z   include-hidden-files: false
2025-12-04T14:54:06.5760825Z env:
2025-12-04T14:54:06.5760940Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:06.5761093Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:06.5761294Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:06.5761480Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:06.5762012Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:06.5762525Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:06.5762658Z   AWS_REGION: us-east-1
2025-12-04T14:54:06.5762834Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:06.5763007Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:06.5765044Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:06.5765234Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:06.5765444Z ##[endgroup]
2025-12-04T14:54:07.0034612Z With the provided path, there will be 902 files uploaded
2025-12-04T14:54:07.0037296Z Artifact name is valid!
2025-12-04T14:54:07.0038093Z Root directory input is valid!
2025-12-04T14:54:07.2367571Z Beginning upload of artifact content to blob storage
2025-12-04T14:54:07.9597498Z Uploaded bytes 660897
2025-12-04T14:54:08.0242320Z Finished uploading artifact content to blob storage!
2025-12-04T14:54:08.0243455Z SHA256 digest of uploaded artifact zip is 9df397243e473d89f9ed0bcea25f9140f0fcfde6f60e107c17d73b6b7d56ec30
2025-12-04T14:54:08.0244153Z Finalizing artifact upload
2025-12-04T14:54:08.1781505Z Artifact test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip.zip successfully finalized. Artifact ID 4765527299
2025-12-04T14:54:08.1782683Z Artifact test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip has been successfully uploaded! Final size is 660897 bytes. Artifact ID is 4765527299
2025-12-04T14:54:08.1786843Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/19922798714/artifacts/4765527299
2025-12-04T14:54:08.1941449Z ##[group]Run actions/upload-artifact@v4
2025-12-04T14:54:08.1941690Z with:
2025-12-04T14:54:08.1941886Z   name: logs-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip
2025-12-04T14:54:08.1942105Z   retention-days: 14
2025-12-04T14:54:08.1942234Z   if-no-files-found: ignore
2025-12-04T14:54:08.1942367Z   path: usage_log.txt
test/**/*.log

2025-12-04T14:54:08.1942505Z   compression-level: 6
2025-12-04T14:54:08.1942621Z   overwrite: false
2025-12-04T14:54:08.1942743Z   include-hidden-files: false
2025-12-04T14:54:08.1942868Z env:
2025-12-04T14:54:08.1942971Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:08.1943120Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:08.1943438Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:08.1943619Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:08.1944153Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:08.1944660Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:08.1944788Z   AWS_REGION: us-east-1
2025-12-04T14:54:08.1944954Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:08.1945125Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:08.1947209Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:08.1947401Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:08.1947596Z ##[endgroup]
2025-12-04T14:54:08.6030642Z Multiple search paths detected. Calculating the least common ancestor of all paths
2025-12-04T14:54:08.6031535Z The least common ancestor is /home/runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2025-12-04T14:54:08.6031836Z With the provided path, there will be 114 files uploaded
2025-12-04T14:54:08.6034177Z Artifact name is valid!
2025-12-04T14:54:08.6034767Z Root directory input is valid!
2025-12-04T14:54:08.8794072Z Beginning upload of artifact content to blob storage
2025-12-04T14:54:09.4128307Z Uploaded bytes 394140
2025-12-04T14:54:09.4817589Z Finished uploading artifact content to blob storage!
2025-12-04T14:54:09.4818993Z SHA256 digest of uploaded artifact zip is bc27956b19bede636f0920aaa7ca6306e129b61f2de6beddfa998c28e621db2d
2025-12-04T14:54:09.4819650Z Finalizing artifact upload
2025-12-04T14:54:09.6238873Z Artifact logs-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip.zip successfully finalized. Artifact ID 4765527612
2025-12-04T14:54:09.6240273Z Artifact logs-runattempt1-test-distributed-3-3-linux.rocm.gpu.gfx942.4.b_57117547539.zip has been successfully uploaded! Final size is 394140 bytes. Artifact ID is 4765527612
2025-12-04T14:54:09.6244709Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/19922798714/artifacts/4765527612
2025-12-04T14:54:09.6351217Z ##[group]Run # shellcheck disable=SC2156
2025-12-04T14:54:09.6351425Z # shellcheck disable=SC2156
2025-12-04T14:54:09.6351675Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2025-12-04T14:54:09.6356102Z shell: /usr/bin/bash -e {0}
2025-12-04T14:54:09.6356232Z env:
2025-12-04T14:54:09.6356344Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:09.6356515Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:09.6356721Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:09.6356981Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:09.6357554Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:09.6358189Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:09.6358389Z   AWS_REGION: us-east-1
2025-12-04T14:54:09.6358575Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:09.6358755Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:09.6360805Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:09.6360992Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:09.6361182Z ##[endgroup]
2025-12-04T14:54:09.7697659Z ##[group]Run actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
2025-12-04T14:54:09.7697856Z with:
2025-12-04T14:54:09.7698010Z   name: coredumps-distributed-3-3-linux.rocm.gpu.gfx942.4.b
2025-12-04T14:54:09.7698190Z   retention-days: 14
2025-12-04T14:54:09.7698312Z   if-no-files-found: ignore
2025-12-04T14:54:09.7698439Z   path: ./**/core.[1-9]*
2025-12-04T14:54:09.7698562Z   compression-level: 6
2025-12-04T14:54:09.7698685Z   overwrite: false
2025-12-04T14:54:09.7698807Z   include-hidden-files: false
2025-12-04T14:54:09.7698934Z env:
2025-12-04T14:54:09.7699038Z   GIT_DEFAULT_BRANCH: main
2025-12-04T14:54:09.7699201Z   RUNNER_ARTIFACT_DIR: /home/runner/_work/_temp/artifacts
2025-12-04T14:54:09.7699403Z   RUNNER_TEST_RESULTS_DIR: /home/runner/_work/_temp/test-results
2025-12-04T14:54:09.7699592Z   RUNNER_DOCS_DIR: /home/runner/_work/_temp/docs
2025-12-04T14:54:09.7700150Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --group-add 110 --device /dev/dri/renderD128 --device /dev/dri/renderD136 --device /dev/dri/renderD144 --device /dev/dri/renderD152 --group-add video --group-add 109 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-12-04T14:54:09.7700643Z   AWS_DEFAULT_REGION: us-east-1
2025-12-04T14:54:09.7700768Z   AWS_REGION: us-east-1
2025-12-04T14:54:09.7700905Z   AWS_ACCESS_KEY_ID: ***
2025-12-04T14:54:09.7701068Z   AWS_SECRET_ACCESS_KEY: ***
2025-12-04T14:54:09.7703091Z   AWS_SESSION_TOKEN: ***
2025-12-04T14:54:09.7703272Z   CONTAINER_NAME: 4690fd9c8512d359282acf92b95f5cea70dbd05cacd2fb6274a8286df117fbca
2025-12-04T14:54:09.7703463Z ##[endgroup]
2025-12-04T14:54:13.6192590Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded.
2025-12-04T14:54:13.6357838Z Post job cleanup.
2025-12-04T14:54:13.6370960Z Post job cleanup.
2025-12-04T14:54:13.6556360Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-12-04T14:54:13.6736179Z Post job cleanup.
2025-12-04T14:54:13.7336427Z Post job cleanup.
2025-12-04T14:54:13.7356029Z Post job cleanup.
2025-12-04T14:54:13.7823460Z [command]/usr/bin/git version
2025-12-04T14:54:13.7852542Z git version 2.52.0
2025-12-04T14:54:13.7875985Z Copying '/home/runner/.gitconfig' to '/home/runner/_work/_temp/29c6f7f2-d3bc-46da-88fc-cd9626a7a17b/.gitconfig'
2025-12-04T14:54:13.7883074Z Temporarily overriding HOME='/home/runner/_work/_temp/29c6f7f2-d3bc-46da-88fc-cd9626a7a17b' before making global git config changes
2025-12-04T14:54:13.7883565Z Adding repository directory to the temporary git global config as a safe directory
2025-12-04T14:54:13.7885783Z [command]/usr/bin/git config --global --add safe.directory /home/runner/_work/pytorch/pytorch
2025-12-04T14:54:13.7914023Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-12-04T14:54:13.7932615Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-12-04T14:54:13.8140870Z Entering 'android/libs/fbjni'
2025-12-04T14:54:13.8162922Z Entering 'third_party/FP16'
2025-12-04T14:54:13.8190021Z Entering 'third_party/FXdiv'
2025-12-04T14:54:13.8225970Z Entering 'third_party/NNPACK'
2025-12-04T14:54:13.8250471Z Entering 'third_party/NVTX'
2025-12-04T14:54:13.8275721Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T14:54:13.8304231Z Entering 'third_party/XNNPACK'
2025-12-04T14:54:13.8336720Z Entering 'third_party/aiter'
2025-12-04T14:54:13.8366073Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T14:54:13.8417434Z Entering 'third_party/benchmark'
2025-12-04T14:54:13.8441579Z Entering 'third_party/composable_kernel'
2025-12-04T14:54:13.8467593Z Entering 'third_party/cpp-httplib'
2025-12-04T14:54:13.8492910Z Entering 'third_party/cpuinfo'
2025-12-04T14:54:13.8516297Z Entering 'third_party/cudnn_frontend'
2025-12-04T14:54:13.8541689Z Entering 'third_party/cutlass'
2025-12-04T14:54:13.8570981Z Entering 'third_party/fbgemm'
2025-12-04T14:54:13.8598157Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T14:54:13.8624831Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T14:54:13.8655266Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T14:54:13.8678442Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T14:54:13.8720696Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T14:54:13.8757470Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T14:54:13.8782371Z Entering 'third_party/fbgemm/external/json'
2025-12-04T14:54:13.8813379Z Entering 'third_party/flash-attention'
2025-12-04T14:54:13.8851766Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T14:54:13.8883865Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T14:54:13.8911916Z Entering 'third_party/flatbuffers'
2025-12-04T14:54:13.8949937Z Entering 'third_party/fmt'
2025-12-04T14:54:13.8986261Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T14:54:13.9016220Z Entering 'third_party/gloo'
2025-12-04T14:54:13.9040679Z Entering 'third_party/googletest'
2025-12-04T14:54:13.9063047Z Entering 'third_party/ideep'
2025-12-04T14:54:13.9090218Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T14:54:13.9125092Z Entering 'third_party/ittapi'
2025-12-04T14:54:13.9152325Z Entering 'third_party/kineto'
2025-12-04T14:54:13.9179427Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T14:54:13.9219901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T14:54:13.9250034Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T14:54:13.9283877Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T14:54:13.9313455Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T14:54:13.9344846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T14:54:13.9379124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T14:54:13.9411438Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T14:54:13.9436938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T14:54:13.9467297Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T14:54:13.9498514Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T14:54:13.9531001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:13.9560679Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:13.9596693Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T14:54:13.9625825Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T14:54:13.9652894Z Entering 'third_party/kleidiai'
2025-12-04T14:54:13.9679683Z Entering 'third_party/mimalloc'
2025-12-04T14:54:13.9703985Z Entering 'third_party/nlohmann'
2025-12-04T14:54:13.9731183Z Entering 'third_party/onnx'
2025-12-04T14:54:13.9770696Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T14:54:13.9805499Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T14:54:13.9842307Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T14:54:13.9865998Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T14:54:13.9894153Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T14:54:13.9922776Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T14:54:13.9948218Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T14:54:13.9977176Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T14:54:14.0006466Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T14:54:14.0030971Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:14.0062324Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:14.0089327Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T14:54:14.0124202Z Entering 'third_party/pocketfft'
2025-12-04T14:54:14.0150709Z Entering 'third_party/protobuf'
2025-12-04T14:54:14.0179110Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T14:54:14.0204158Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T14:54:14.0229221Z Entering 'third_party/psimd'
2025-12-04T14:54:14.0255971Z Entering 'third_party/pthreadpool'
2025-12-04T14:54:14.0281389Z Entering 'third_party/pybind11'
2025-12-04T14:54:14.0307155Z Entering 'third_party/python-peachpy'
2025-12-04T14:54:14.0336111Z Entering 'third_party/sleef'
2025-12-04T14:54:14.0362657Z Entering 'third_party/tensorpipe'
2025-12-04T14:54:14.0389388Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T14:54:14.0419037Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T14:54:14.0442386Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T14:54:14.0470411Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T14:54:14.0501259Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T14:54:14.0555591Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-12-04T14:54:14.0571465Z http.https://github.com/.extraheader
2025-12-04T14:54:14.0582858Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-12-04T14:54:14.0606099Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-12-04T14:54:14.0772143Z Entering 'android/libs/fbjni'
2025-12-04T14:54:14.0793641Z http.https://github.com/.extraheader
2025-12-04T14:54:14.0813805Z Entering 'third_party/FP16'
2025-12-04T14:54:14.0830667Z http.https://github.com/.extraheader
2025-12-04T14:54:14.0849885Z Entering 'third_party/FXdiv'
2025-12-04T14:54:14.0867169Z http.https://github.com/.extraheader
2025-12-04T14:54:14.0884591Z Entering 'third_party/NNPACK'
2025-12-04T14:54:14.0901427Z http.https://github.com/.extraheader
2025-12-04T14:54:14.0927993Z Entering 'third_party/NVTX'
2025-12-04T14:54:14.0942680Z http.https://github.com/.extraheader
2025-12-04T14:54:14.0959426Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T14:54:14.0973628Z http.https://github.com/.extraheader
2025-12-04T14:54:14.0992486Z Entering 'third_party/XNNPACK'
2025-12-04T14:54:14.1005109Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1035190Z Entering 'third_party/aiter'
2025-12-04T14:54:14.1053269Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1076513Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T14:54:14.1089886Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1112305Z Entering 'third_party/benchmark'
2025-12-04T14:54:14.1135973Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1151919Z Entering 'third_party/composable_kernel'
2025-12-04T14:54:14.1165552Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1188118Z Entering 'third_party/cpp-httplib'
2025-12-04T14:54:14.1200813Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1226721Z Entering 'third_party/cpuinfo'
2025-12-04T14:54:14.1240288Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1258935Z Entering 'third_party/cudnn_frontend'
2025-12-04T14:54:14.1271957Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1295890Z Entering 'third_party/cutlass'
2025-12-04T14:54:14.1312626Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1336029Z Entering 'third_party/fbgemm'
2025-12-04T14:54:14.1357000Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1377497Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T14:54:14.1395463Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1414775Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T14:54:14.1431373Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1453813Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T14:54:14.1473266Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1492783Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T14:54:14.1509190Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1530250Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T14:54:14.1544237Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1567208Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T14:54:14.1582792Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1597953Z Entering 'third_party/fbgemm/external/json'
2025-12-04T14:54:14.1613474Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1633073Z Entering 'third_party/flash-attention'
2025-12-04T14:54:14.1650366Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1672031Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T14:54:14.1686153Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1709851Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T14:54:14.1724234Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1747646Z Entering 'third_party/flatbuffers'
2025-12-04T14:54:14.1763696Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1784765Z Entering 'third_party/fmt'
2025-12-04T14:54:14.1801629Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1821524Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T14:54:14.1837517Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1854857Z Entering 'third_party/gloo'
2025-12-04T14:54:14.1875341Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1893120Z Entering 'third_party/googletest'
2025-12-04T14:54:14.1910054Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1928843Z Entering 'third_party/ideep'
2025-12-04T14:54:14.1946355Z http.https://github.com/.extraheader
2025-12-04T14:54:14.1966061Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T14:54:14.1983637Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2006829Z Entering 'third_party/ittapi'
2025-12-04T14:54:14.2024458Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2041579Z Entering 'third_party/kineto'
2025-12-04T14:54:14.2060484Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2080450Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T14:54:14.2093334Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2111423Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T14:54:14.2124542Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2146190Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T14:54:14.2160872Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2184680Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T14:54:14.2201978Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2218141Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T14:54:14.2232343Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2262428Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T14:54:14.2275845Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2295317Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T14:54:14.2310510Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2341678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T14:54:14.2360162Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2381715Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T14:54:14.2396683Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2413611Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T14:54:14.2427230Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2446854Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T14:54:14.2463544Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2479315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:14.2495559Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2519522Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:14.2537346Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2568776Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T14:54:14.2583345Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2605756Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T14:54:14.2618883Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2640080Z Entering 'third_party/kleidiai'
2025-12-04T14:54:14.2654949Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2686297Z Entering 'third_party/mimalloc'
2025-12-04T14:54:14.2702364Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2720418Z Entering 'third_party/nlohmann'
2025-12-04T14:54:14.2737927Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2759150Z Entering 'third_party/onnx'
2025-12-04T14:54:14.2774525Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2800181Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T14:54:14.2816169Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2835877Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T14:54:14.2849815Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2876612Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T14:54:14.2891862Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2915979Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T14:54:14.2936046Z http.https://github.com/.extraheader
2025-12-04T14:54:14.2957889Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T14:54:14.2975189Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3001733Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T14:54:14.3015927Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3038338Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T14:54:14.3053491Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3071303Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T14:54:14.3089934Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3107661Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T14:54:14.3125458Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3143599Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:14.3158280Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3181840Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:14.3196291Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3214689Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T14:54:14.3233716Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3265501Z Entering 'third_party/pocketfft'
2025-12-04T14:54:14.3284427Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3303789Z Entering 'third_party/protobuf'
2025-12-04T14:54:14.3317251Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3342851Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T14:54:14.3360372Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3383649Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T14:54:14.3401109Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3426161Z Entering 'third_party/psimd'
2025-12-04T14:54:14.3442098Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3460482Z Entering 'third_party/pthreadpool'
2025-12-04T14:54:14.3474543Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3494848Z Entering 'third_party/pybind11'
2025-12-04T14:54:14.3508448Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3530838Z Entering 'third_party/python-peachpy'
2025-12-04T14:54:14.3548331Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3565729Z Entering 'third_party/sleef'
2025-12-04T14:54:14.3583167Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3600438Z Entering 'third_party/tensorpipe'
2025-12-04T14:54:14.3614781Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3637624Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T14:54:14.3652897Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3681596Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T14:54:14.3701727Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3720099Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T14:54:14.3736396Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3753888Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T14:54:14.3772716Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3793734Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T14:54:14.3809732Z http.https://github.com/.extraheader
2025-12-04T14:54:14.3848968Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.3880863Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
2025-12-04T14:54:14.4070748Z Entering 'android/libs/fbjni'
2025-12-04T14:54:14.4086730Z file:/home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-12-04T14:54:14.4096832Z Entering 'third_party/FP16'
2025-12-04T14:54:14.4111846Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-12-04T14:54:14.4123039Z Entering 'third_party/FXdiv'
2025-12-04T14:54:14.4135315Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-12-04T14:54:14.4145861Z Entering 'third_party/NNPACK'
2025-12-04T14:54:14.4157520Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-12-04T14:54:14.4169099Z Entering 'third_party/NVTX'
2025-12-04T14:54:14.4180154Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-12-04T14:54:14.4189034Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T14:54:14.4199174Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-12-04T14:54:14.4212785Z Entering 'third_party/XNNPACK'
2025-12-04T14:54:14.4228573Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-12-04T14:54:14.4244625Z Entering 'third_party/aiter'
2025-12-04T14:54:14.4262565Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-12-04T14:54:14.4274087Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T14:54:14.4284231Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-12-04T14:54:14.4298768Z Entering 'third_party/benchmark'
2025-12-04T14:54:14.4308944Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T14:54:14.4317655Z Entering 'third_party/composable_kernel'
2025-12-04T14:54:14.4327478Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-12-04T14:54:14.4342893Z Entering 'third_party/cpp-httplib'
2025-12-04T14:54:14.4353413Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-12-04T14:54:14.4362201Z Entering 'third_party/cpuinfo'
2025-12-04T14:54:14.4372202Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-12-04T14:54:14.4380659Z Entering 'third_party/cudnn_frontend'
2025-12-04T14:54:14.4393197Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-12-04T14:54:14.4401900Z Entering 'third_party/cutlass'
2025-12-04T14:54:14.4416647Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-12-04T14:54:14.4432810Z Entering 'third_party/fbgemm'
2025-12-04T14:54:14.4446578Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-12-04T14:54:14.4456263Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T14:54:14.4474500Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-12-04T14:54:14.4483381Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T14:54:14.4492461Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-12-04T14:54:14.4504377Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T14:54:14.4516032Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-12-04T14:54:14.4524069Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T14:54:14.4538564Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-12-04T14:54:14.4551859Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T14:54:14.4562608Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-12-04T14:54:14.4571565Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T14:54:14.4597187Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-12-04T14:54:14.4597926Z Entering 'third_party/fbgemm/external/json'
2025-12-04T14:54:14.4607336Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-12-04T14:54:14.4624791Z Entering 'third_party/flash-attention'
2025-12-04T14:54:14.4637807Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-12-04T14:54:14.4650863Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T14:54:14.4667084Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-12-04T14:54:14.4682228Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T14:54:14.4695552Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-12-04T14:54:14.4711974Z Entering 'third_party/flatbuffers'
2025-12-04T14:54:14.4722309Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-12-04T14:54:14.4734290Z Entering 'third_party/fmt'
2025-12-04T14:54:14.4743809Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-12-04T14:54:14.4751915Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T14:54:14.4762618Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-12-04T14:54:14.4772500Z Entering 'third_party/gloo'
2025-12-04T14:54:14.4781996Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-12-04T14:54:14.4795281Z Entering 'third_party/googletest'
2025-12-04T14:54:14.4804845Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:14.4813307Z Entering 'third_party/ideep'
2025-12-04T14:54:14.4823044Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-12-04T14:54:14.4833391Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T14:54:14.4842885Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-12-04T14:54:14.4854292Z Entering 'third_party/ittapi'
2025-12-04T14:54:14.4864792Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-12-04T14:54:14.4873886Z Entering 'third_party/kineto'
2025-12-04T14:54:14.4884453Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-12-04T14:54:14.4894725Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T14:54:14.4909962Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-12-04T14:54:14.4919936Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T14:54:14.4929865Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-12-04T14:54:14.4940405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T14:54:14.4950093Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-12-04T14:54:14.4957851Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T14:54:14.4978102Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-12-04T14:54:14.4987265Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T14:54:14.5012322Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-12-04T14:54:14.5021574Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T14:54:14.5030728Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-12-04T14:54:14.5043482Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T14:54:14.5055054Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-12-04T14:54:14.5063110Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T14:54:14.5072460Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:14.5085659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T14:54:14.5098523Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-12-04T14:54:14.5111949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T14:54:14.5123216Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-12-04T14:54:14.5133507Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T14:54:14.5143050Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T14:54:14.5158826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:14.5172113Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T14:54:14.5186539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:14.5200526Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T14:54:14.5215488Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T14:54:14.5227235Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-12-04T14:54:14.5237038Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T14:54:14.5248842Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:14.5260339Z Entering 'third_party/kleidiai'
2025-12-04T14:54:14.5272577Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-12-04T14:54:14.5283172Z Entering 'third_party/mimalloc'
2025-12-04T14:54:14.5295139Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-12-04T14:54:14.5304860Z Entering 'third_party/nlohmann'
2025-12-04T14:54:14.5317135Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-12-04T14:54:14.5326698Z Entering 'third_party/onnx'
2025-12-04T14:54:14.5339505Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-12-04T14:54:14.5356079Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T14:54:14.5367270Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T14:54:14.5379962Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T14:54:14.5389746Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-12-04T14:54:14.5398853Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T14:54:14.5411229Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T14:54:14.5420562Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T14:54:14.5431109Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:14.5442054Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T14:54:14.5458195Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-12-04T14:54:14.5467772Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T14:54:14.5479335Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-12-04T14:54:14.5489730Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T14:54:14.5501234Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-12-04T14:54:14.5511317Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T14:54:14.5521661Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-12-04T14:54:14.5530751Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T14:54:14.5545535Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T14:54:14.5554630Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:14.5568536Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T14:54:14.5579309Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:14.5589910Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T14:54:14.5600975Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T14:54:14.5612563Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-12-04T14:54:14.5629757Z Entering 'third_party/pocketfft'
2025-12-04T14:54:14.5640148Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-12-04T14:54:14.5648237Z Entering 'third_party/protobuf'
2025-12-04T14:54:14.5658166Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-12-04T14:54:14.5667707Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T14:54:14.5679087Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T14:54:14.5688666Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T14:54:14.5703384Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:14.5713593Z Entering 'third_party/psimd'
2025-12-04T14:54:14.5724967Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-12-04T14:54:14.5732957Z Entering 'third_party/pthreadpool'
2025-12-04T14:54:14.5743014Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-12-04T14:54:14.5751611Z Entering 'third_party/pybind11'
2025-12-04T14:54:14.5761167Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T14:54:14.5770807Z Entering 'third_party/python-peachpy'
2025-12-04T14:54:14.5780589Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-12-04T14:54:14.5790172Z Entering 'third_party/sleef'
2025-12-04T14:54:14.5799487Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-12-04T14:54:14.5808060Z Entering 'third_party/tensorpipe'
2025-12-04T14:54:14.5818430Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-12-04T14:54:14.5826918Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T14:54:14.5839440Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:14.5849338Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T14:54:14.5859935Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-12-04T14:54:14.5869080Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T14:54:14.5880110Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-12-04T14:54:14.5889483Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T14:54:14.5900996Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T14:54:14.5909735Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T14:54:14.5921106Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-12-04T14:54:14.5948961Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.5968509Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.5986845Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6001797Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6017722Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6036840Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6052919Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6066908Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6082588Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6097051Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6113755Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6128029Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6142594Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6156503Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6173731Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6189370Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6204131Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6218108Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6234885Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6249942Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6264157Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6279913Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6293447Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6311420Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6327843Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6342245Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6356714Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6370323Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6389129Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6403609Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6418327Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6432664Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6448081Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6464210Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6478291Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6492278Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6507082Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6522642Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6538081Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6560151Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6576244Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6589940Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6604854Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6619364Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6633473Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6648303Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6665143Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6680642Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6697837Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6716066Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6730732Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6744779Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6759804Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6774921Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6789795Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6805091Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6821432Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6837285Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6854033Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6870861Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6886590Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6906446Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6924911Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6939941Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6955518Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6971969Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.6988137Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7002996Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7018623Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7033687Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7048685Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7062553Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7078054Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7093395Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7108448Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7123066Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7142274Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7159004Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7173282Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7188782Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7202884Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:14.7301330Z Post job cleanup.
2025-12-04T14:54:14.7745701Z [command]/usr/bin/git version
2025-12-04T14:54:14.7765677Z git version 2.52.0
2025-12-04T14:54:14.7782130Z Copying '/home/runner/.gitconfig' to '/home/runner/_work/_temp/701b3535-359f-4bb1-88d7-25da90cc051b/.gitconfig'
2025-12-04T14:54:14.7787277Z Temporarily overriding HOME='/home/runner/_work/_temp/701b3535-359f-4bb1-88d7-25da90cc051b' before making global git config changes
2025-12-04T14:54:14.7787809Z Adding repository directory to the temporary git global config as a safe directory
2025-12-04T14:54:14.7789083Z [command]/usr/bin/git config --global --add safe.directory /home/runner/_work/pytorch/pytorch
2025-12-04T14:54:14.7810935Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-12-04T14:54:14.7834207Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-12-04T14:54:14.8028184Z Entering 'android/libs/fbjni'
2025-12-04T14:54:14.8049566Z Entering 'third_party/FP16'
2025-12-04T14:54:14.8071642Z Entering 'third_party/FXdiv'
2025-12-04T14:54:14.8101445Z Entering 'third_party/NNPACK'
2025-12-04T14:54:14.8125742Z Entering 'third_party/NVTX'
2025-12-04T14:54:14.8149255Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T14:54:14.8174171Z Entering 'third_party/XNNPACK'
2025-12-04T14:54:14.8202017Z Entering 'third_party/aiter'
2025-12-04T14:54:14.8226564Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T14:54:14.8255581Z Entering 'third_party/benchmark'
2025-12-04T14:54:14.8278564Z Entering 'third_party/composable_kernel'
2025-12-04T14:54:14.8304033Z Entering 'third_party/cpp-httplib'
2025-12-04T14:54:14.8327065Z Entering 'third_party/cpuinfo'
2025-12-04T14:54:14.8350016Z Entering 'third_party/cudnn_frontend'
2025-12-04T14:54:14.8376902Z Entering 'third_party/cutlass'
2025-12-04T14:54:14.8404219Z Entering 'third_party/fbgemm'
2025-12-04T14:54:14.8430994Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T14:54:14.8463408Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T14:54:14.8497934Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T14:54:14.8522116Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T14:54:14.8555801Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T14:54:14.8580868Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T14:54:14.8610380Z Entering 'third_party/fbgemm/external/json'
2025-12-04T14:54:14.8640552Z Entering 'third_party/flash-attention'
2025-12-04T14:54:14.8669495Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T14:54:14.8697621Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T14:54:14.8726434Z Entering 'third_party/flatbuffers'
2025-12-04T14:54:14.8753393Z Entering 'third_party/fmt'
2025-12-04T14:54:14.8785526Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T14:54:14.8809191Z Entering 'third_party/gloo'
2025-12-04T14:54:14.8833162Z Entering 'third_party/googletest'
2025-12-04T14:54:14.8860301Z Entering 'third_party/ideep'
2025-12-04T14:54:14.8883236Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T14:54:14.8912737Z Entering 'third_party/ittapi'
2025-12-04T14:54:14.8934880Z Entering 'third_party/kineto'
2025-12-04T14:54:14.8956716Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T14:54:14.8986557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T14:54:14.9011881Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T14:54:14.9038968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T14:54:14.9065923Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T14:54:14.9098110Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T14:54:14.9129386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T14:54:14.9150632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T14:54:14.9171640Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T14:54:14.9192743Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T14:54:14.9221171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T14:54:14.9244806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:14.9271770Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:14.9305895Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T14:54:14.9340510Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T14:54:14.9371454Z Entering 'third_party/kleidiai'
2025-12-04T14:54:14.9405399Z Entering 'third_party/mimalloc'
2025-12-04T14:54:14.9437633Z Entering 'third_party/nlohmann'
2025-12-04T14:54:14.9462013Z Entering 'third_party/onnx'
2025-12-04T14:54:14.9499505Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T14:54:14.9526206Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T14:54:14.9553513Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T14:54:14.9581607Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T14:54:14.9606278Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T14:54:14.9631435Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T14:54:14.9654387Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T14:54:14.9678598Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T14:54:14.9706829Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T14:54:14.9736366Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:14.9765471Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:14.9793151Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T14:54:14.9825864Z Entering 'third_party/pocketfft'
2025-12-04T14:54:14.9853360Z Entering 'third_party/protobuf'
2025-12-04T14:54:14.9879698Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T14:54:14.9905075Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T14:54:14.9935008Z Entering 'third_party/psimd'
2025-12-04T14:54:14.9959223Z Entering 'third_party/pthreadpool'
2025-12-04T14:54:14.9982679Z Entering 'third_party/pybind11'
2025-12-04T14:54:15.0006495Z Entering 'third_party/python-peachpy'
2025-12-04T14:54:15.0037128Z Entering 'third_party/sleef'
2025-12-04T14:54:15.0059607Z Entering 'third_party/tensorpipe'
2025-12-04T14:54:15.0094096Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T14:54:15.0125565Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T14:54:15.0156350Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T14:54:15.0186825Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T14:54:15.0214663Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T14:54:15.0267728Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-12-04T14:54:15.0291357Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-12-04T14:54:15.0479356Z Entering 'android/libs/fbjni'
2025-12-04T14:54:15.0513105Z Entering 'third_party/FP16'
2025-12-04T14:54:15.0544219Z Entering 'third_party/FXdiv'
2025-12-04T14:54:15.0568817Z Entering 'third_party/NNPACK'
2025-12-04T14:54:15.0597864Z Entering 'third_party/NVTX'
2025-12-04T14:54:15.0623324Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T14:54:15.0647465Z Entering 'third_party/XNNPACK'
2025-12-04T14:54:15.0676946Z Entering 'third_party/aiter'
2025-12-04T14:54:15.0701666Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T14:54:15.0733693Z Entering 'third_party/benchmark'
2025-12-04T14:54:15.0757462Z Entering 'third_party/composable_kernel'
2025-12-04T14:54:15.0788471Z Entering 'third_party/cpp-httplib'
2025-12-04T14:54:15.0811222Z Entering 'third_party/cpuinfo'
2025-12-04T14:54:15.0838322Z Entering 'third_party/cudnn_frontend'
2025-12-04T14:54:15.0858931Z Entering 'third_party/cutlass'
2025-12-04T14:54:15.0883763Z Entering 'third_party/fbgemm'
2025-12-04T14:54:15.0910367Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T14:54:15.0933165Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T14:54:15.0966297Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T14:54:15.0991507Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T14:54:15.1018574Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T14:54:15.1040050Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T14:54:15.1059264Z Entering 'third_party/fbgemm/external/json'
2025-12-04T14:54:15.1080345Z Entering 'third_party/flash-attention'
2025-12-04T14:54:15.1102502Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T14:54:15.1132552Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T14:54:15.1167658Z Entering 'third_party/flatbuffers'
2025-12-04T14:54:15.1198545Z Entering 'third_party/fmt'
2025-12-04T14:54:15.1224763Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T14:54:15.1247860Z Entering 'third_party/gloo'
2025-12-04T14:54:15.1268512Z Entering 'third_party/googletest'
2025-12-04T14:54:15.1291507Z Entering 'third_party/ideep'
2025-12-04T14:54:15.1314306Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T14:54:15.1345063Z Entering 'third_party/ittapi'
2025-12-04T14:54:15.1365479Z Entering 'third_party/kineto'
2025-12-04T14:54:15.1386404Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T14:54:15.1417783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T14:54:15.1450631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T14:54:15.1482493Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T14:54:15.1512879Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T14:54:15.1540549Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T14:54:15.1564325Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T14:54:15.1586048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T14:54:15.1613077Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T14:54:15.1639177Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T14:54:15.1666883Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T14:54:15.1696565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:15.1723644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:15.1761172Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T14:54:15.1786337Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T14:54:15.1826115Z Entering 'third_party/kleidiai'
2025-12-04T14:54:15.1840696Z Entering 'third_party/mimalloc'
2025-12-04T14:54:15.1867877Z Entering 'third_party/nlohmann'
2025-12-04T14:54:15.1900961Z Entering 'third_party/onnx'
2025-12-04T14:54:15.1935132Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T14:54:15.1962677Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T14:54:15.1985497Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T14:54:15.2015889Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T14:54:15.2036584Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T14:54:15.2061374Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T14:54:15.2087336Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T14:54:15.2113183Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T14:54:15.2137039Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T14:54:15.2164301Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:15.2186113Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:15.2212742Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T14:54:15.2248995Z Entering 'third_party/pocketfft'
2025-12-04T14:54:15.2277080Z Entering 'third_party/protobuf'
2025-12-04T14:54:15.2302350Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T14:54:15.2333457Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T14:54:15.2360027Z Entering 'third_party/psimd'
2025-12-04T14:54:15.2390712Z Entering 'third_party/pthreadpool'
2025-12-04T14:54:15.2415659Z Entering 'third_party/pybind11'
2025-12-04T14:54:15.2436928Z Entering 'third_party/python-peachpy'
2025-12-04T14:54:15.2458363Z Entering 'third_party/sleef'
2025-12-04T14:54:15.2489200Z Entering 'third_party/tensorpipe'
2025-12-04T14:54:15.2519352Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T14:54:15.2549714Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T14:54:15.2574243Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T14:54:15.2600096Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T14:54:15.2630372Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T14:54:15.2679168Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.2702717Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
2025-12-04T14:54:15.2882931Z Entering 'android/libs/fbjni'
2025-12-04T14:54:15.2897604Z file:/home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-12-04T14:54:15.2906929Z Entering 'third_party/FP16'
2025-12-04T14:54:15.2924356Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-12-04T14:54:15.2935104Z Entering 'third_party/FXdiv'
2025-12-04T14:54:15.2947994Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-12-04T14:54:15.2956383Z Entering 'third_party/NNPACK'
2025-12-04T14:54:15.2968787Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-12-04T14:54:15.2978409Z Entering 'third_party/NVTX'
2025-12-04T14:54:15.2988116Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-12-04T14:54:15.2997073Z Entering 'third_party/VulkanMemoryAllocator'
2025-12-04T14:54:15.3006913Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-12-04T14:54:15.3020492Z Entering 'third_party/XNNPACK'
2025-12-04T14:54:15.3035893Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-12-04T14:54:15.3051269Z Entering 'third_party/aiter'
2025-12-04T14:54:15.3063843Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-12-04T14:54:15.3075338Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-12-04T14:54:15.3085353Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-12-04T14:54:15.3101535Z Entering 'third_party/benchmark'
2025-12-04T14:54:15.3111736Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T14:54:15.3125917Z Entering 'third_party/composable_kernel'
2025-12-04T14:54:15.3140467Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-12-04T14:54:15.3151820Z Entering 'third_party/cpp-httplib'
2025-12-04T14:54:15.3162606Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-12-04T14:54:15.3170685Z Entering 'third_party/cpuinfo'
2025-12-04T14:54:15.3182210Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-12-04T14:54:15.3193130Z Entering 'third_party/cudnn_frontend'
2025-12-04T14:54:15.3203807Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-12-04T14:54:15.3212715Z Entering 'third_party/cutlass'
2025-12-04T14:54:15.3224516Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-12-04T14:54:15.3236966Z Entering 'third_party/fbgemm'
2025-12-04T14:54:15.3246670Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-12-04T14:54:15.3262332Z Entering 'third_party/fbgemm/external/asmjit'
2025-12-04T14:54:15.3275373Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-12-04T14:54:15.3285425Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-12-04T14:54:15.3295216Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-12-04T14:54:15.3309909Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-12-04T14:54:15.3321133Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-12-04T14:54:15.3329380Z Entering 'third_party/fbgemm/external/cutlass'
2025-12-04T14:54:15.3344914Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-12-04T14:54:15.3360693Z Entering 'third_party/fbgemm/external/googletest'
2025-12-04T14:54:15.3370994Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-12-04T14:54:15.3379851Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-12-04T14:54:15.3391244Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-12-04T14:54:15.3401101Z Entering 'third_party/fbgemm/external/json'
2025-12-04T14:54:15.3410569Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-12-04T14:54:15.3421750Z Entering 'third_party/flash-attention'
2025-12-04T14:54:15.3431371Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-12-04T14:54:15.3439652Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-12-04T14:54:15.3448572Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-12-04T14:54:15.3459359Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-12-04T14:54:15.3468198Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-12-04T14:54:15.3480811Z Entering 'third_party/flatbuffers'
2025-12-04T14:54:15.3490569Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-12-04T14:54:15.3507851Z Entering 'third_party/fmt'
2025-12-04T14:54:15.3523636Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-12-04T14:54:15.3534123Z Entering 'third_party/gemmlowp/gemmlowp'
2025-12-04T14:54:15.3548233Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-12-04T14:54:15.3563180Z Entering 'third_party/gloo'
2025-12-04T14:54:15.3573758Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-12-04T14:54:15.3584813Z Entering 'third_party/googletest'
2025-12-04T14:54:15.3597737Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:15.3607671Z Entering 'third_party/ideep'
2025-12-04T14:54:15.3619854Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-12-04T14:54:15.3629173Z Entering 'third_party/ideep/mkl-dnn'
2025-12-04T14:54:15.3643063Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-12-04T14:54:15.3657500Z Entering 'third_party/ittapi'
2025-12-04T14:54:15.3667597Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-12-04T14:54:15.3676579Z Entering 'third_party/kineto'
2025-12-04T14:54:15.3686856Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-12-04T14:54:15.3696218Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-12-04T14:54:15.3713757Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-12-04T14:54:15.3723157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-12-04T14:54:15.3736061Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-12-04T14:54:15.3746086Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-12-04T14:54:15.3759391Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-12-04T14:54:15.3768405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-12-04T14:54:15.3782761Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-12-04T14:54:15.3798548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-12-04T14:54:15.3811546Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-12-04T14:54:15.3820640Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-12-04T14:54:15.3830177Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-12-04T14:54:15.3841008Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-12-04T14:54:15.3853942Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-12-04T14:54:15.3862670Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-12-04T14:54:15.3881859Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:15.3890641Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-12-04T14:54:15.3904681Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-12-04T14:54:15.3916333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-12-04T14:54:15.3925900Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-12-04T14:54:15.3935075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'
2025-12-04T14:54:15.3947064Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T14:54:15.3962156Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:15.3971894Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T14:54:15.3981970Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:15.3999727Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T14:54:15.4012923Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-12-04T14:54:15.4022236Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-12-04T14:54:15.4031066Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-12-04T14:54:15.4045603Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:15.4056898Z Entering 'third_party/kleidiai'
2025-12-04T14:54:15.4072994Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-12-04T14:54:15.4085475Z Entering 'third_party/mimalloc'
2025-12-04T14:54:15.4102351Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-12-04T14:54:15.4113748Z Entering 'third_party/nlohmann'
2025-12-04T14:54:15.4128636Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-12-04T14:54:15.4144539Z Entering 'third_party/onnx'
2025-12-04T14:54:15.4161459Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-12-04T14:54:15.4191039Z Entering 'third_party/onnx/third_party/pybind11'
2025-12-04T14:54:15.4200090Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T14:54:15.4215532Z Entering 'third_party/opentelemetry-cpp'
2025-12-04T14:54:15.4226104Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-12-04T14:54:15.4236418Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-12-04T14:54:15.4247046Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T14:54:15.4256328Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-12-04T14:54:15.4271104Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:15.4279647Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-12-04T14:54:15.4297928Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-12-04T14:54:15.4310667Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-12-04T14:54:15.4323028Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-12-04T14:54:15.4335432Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-12-04T14:54:15.4351152Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-12-04T14:54:15.4359927Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-12-04T14:54:15.4373777Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-12-04T14:54:15.4381251Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-12-04T14:54:15.4392733Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-12-04T14:54:15.4405656Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-12-04T14:54:15.4416235Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-12-04T14:54:15.4426135Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-12-04T14:54:15.4442620Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-12-04T14:54:15.4457759Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-12-04T14:54:15.4469470Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-12-04T14:54:15.4488220Z Entering 'third_party/pocketfft'
2025-12-04T14:54:15.4499230Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-12-04T14:54:15.4507898Z Entering 'third_party/protobuf'
2025-12-04T14:54:15.4519491Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-12-04T14:54:15.4529453Z Entering 'third_party/protobuf/third_party/benchmark'
2025-12-04T14:54:15.4546841Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-12-04T14:54:15.4559659Z Entering 'third_party/protobuf/third_party/googletest'
2025-12-04T14:54:15.4573621Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:15.4584743Z Entering 'third_party/psimd'
2025-12-04T14:54:15.4596981Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-12-04T14:54:15.4604763Z Entering 'third_party/pthreadpool'
2025-12-04T14:54:15.4614589Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-12-04T14:54:15.4623027Z Entering 'third_party/pybind11'
2025-12-04T14:54:15.4637792Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T14:54:15.4647102Z Entering 'third_party/python-peachpy'
2025-12-04T14:54:15.4657548Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-12-04T14:54:15.4666648Z Entering 'third_party/sleef'
2025-12-04T14:54:15.4676640Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-12-04T14:54:15.4685250Z Entering 'third_party/tensorpipe'
2025-12-04T14:54:15.4699580Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-12-04T14:54:15.4708363Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-12-04T14:54:15.4722842Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-12-04T14:54:15.4735884Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-12-04T14:54:15.4753116Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-12-04T14:54:15.4762981Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-12-04T14:54:15.4779673Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-12-04T14:54:15.4790423Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-12-04T14:54:15.4801800Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-12-04T14:54:15.4810920Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-12-04T14:54:15.4824338Z file:/home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-12-04T14:54:15.4861141Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.4886294Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.4903679Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.4921507Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.4941150Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.4960715Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.4981276Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5002554Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5019887Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5037442Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5052933Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5070497Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5086209Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5105532Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5121060Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5141923Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5158113Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5175043Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5190822Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5207551Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5223252Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5243523Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5260733Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5281135Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5302221Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5318157Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5334129Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5348676Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5364753Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5379988Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5394273Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5410082Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5425203Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5442461Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5458085Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5473463Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5489866Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5509043Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5526441Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5542930Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5566558Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5586503Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5604898Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5626848Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5643862Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5664916Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5685398Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5702393Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5719950Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5737294Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5753806Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5770420Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5786457Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5806206Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5823080Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5839930Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5858714Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5876252Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5894379Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5913636Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5934228Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5955085Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5972174Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.5989140Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6005945Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6022546Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6051091Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6068777Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6085805Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6101406Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6119353Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6136237Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6153291Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6170400Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6187683Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6204021Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6220591Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6237390Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6254632Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6271737Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6288474Z [command]/usr/bin/git config --file /home/runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir:
2025-12-04T14:54:15.6411772Z Cleaning up orphan processes