2025-12-04T08:56:27.9815104Z Current runner version: '2.330.0' 2025-12-04T08:56:27.9821469Z Runner name: 'i-0452b6a3e0f4cdcdb' 2025-12-04T08:56:27.9822336Z Runner group name: 'Default' 2025-12-04T08:56:27.9823357Z Machine name: 'ip-10-1-50-12' 2025-12-04T08:56:27.9827133Z ##[group]GITHUB_TOKEN Permissions 2025-12-04T08:56:27.9829699Z Contents: read 2025-12-04T08:56:27.9830446Z Metadata: read 2025-12-04T08:56:27.9831021Z ##[endgroup] 2025-12-04T08:56:27.9833508Z Secret source: Actions 2025-12-04T08:56:27.9834444Z Prepare workflow directory 2025-12-04T08:56:28.0393347Z Prepare all required actions 2025-12-04T08:56:28.0436375Z Getting action download info 2025-12-04T08:56:28.3790306Z Download action repository 'pytorch/test-infra@main' (SHA:39aa74d619174326f4e2fb0e216151c2f29d9ffd) 2025-12-04T08:56:30.7475443Z Download action repository 'pytorch/pytorch@main' (SHA:eabb7ad2128580ef674446027b95bcf4e21e8df3) 2025-12-04T08:56:46.3902869Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-12-04T08:56:46.8207143Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-12-04T08:56:47.0873668Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-12-04T08:56:47.2694236Z Download action repository 'seemethere/download-artifact-s3@1da556a7aa0a088e3153970611f6c432d58e80e6' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T08:56:47.5245553Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T08:56:47.8106455Z Getting action download info 2025-12-04T08:56:47.9329569Z Download action repository 'actions/checkout@v4' (SHA:34e114876b0b11c390a56381ad16ebd13914f8d5) 2025-12-04T08:56:48.1992605Z Getting action download info 2025-12-04T08:56:48.3046475Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-12-04T08:56:48.4884157Z Getting action download info 2025-12-04T08:56:48.6133087Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-12-04T08:56:48.8331566Z Getting action download info 2025-12-04T08:56:49.0040980Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32) 2025-12-04T08:56:49.0044978Z ##[group] Inputs 2025-12-04T08:56:49.0045363Z build-environment: linux-jammy-cuda12.8-py3.10-gcc11 2025-12-04T08:56:49.0056153Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}]} 2025-12-04T08:56:49.0068449Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:56:49.0069455Z sync-tag: 2025-12-04T08:56:49.0070355Z timeout-minutes: 360 2025-12-04T08:56:49.0070664Z use-gha: 2025-12-04T08:56:49.0070907Z dashboard-tag: 2025-12-04T08:56:49.0071199Z s3-bucket: gha-artifacts 2025-12-04T08:56:49.0071520Z aws-role-to-assume: 2025-12-04T08:56:49.0072160Z disable-monitor: false 2025-12-04T08:56:49.0072511Z monitor-log-interval: 5 2025-12-04T08:56:49.0072859Z monitor-data-collect-interval: 1 2025-12-04T08:56:49.0073214Z ##[endgroup] 2025-12-04T08:56:49.0073894Z Complete job name: linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T08:56:49.0784335Z A job started hook has been configured by the self-hosted runner administrator 2025-12-04T08:56:49.0892948Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-12-04T08:56:49.0901977Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:56:49.0902954Z ##[endgroup] 2025-12-04T08:56:50.5719316Z Runner Type: lf.linux.g4dn.12xlarge.nvidia.gpu 2025-12-04T08:56:50.5720048Z Instance Type: g4dn.12xlarge 2025-12-04T08:56:50.5720378Z AMI Name: unknown 2025-12-04T08:56:50.5749583Z AMI ID: ami-08982f1c5bf93d976 2025-12-04T08:56:56.0476382Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-12-04T08:56:56.0476915Z with: 2025-12-04T08:56:56.0477558Z github-secret: *** 2025-12-04T08:56:56.0478394Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-12-04T08:56:56.0479322Z activate-with-label: false 2025-12-04T08:56:56.0479648Z label: with-ssh 2025-12-04T08:56:56.0480053Z remove-existing-keys: true 2025-12-04T08:56:56.0480347Z fail-silently: true 2025-12-04T08:56:56.0480605Z env: 2025-12-04T08:56:56.0480814Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:56:56.0481087Z ##[endgroup] 2025-12-04T08:56:56.1733324Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-12-04T08:56:56.1735056Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-12-04T08:56:56.1903955Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-12-04T08:56:56.1904611Z with: 2025-12-04T08:56:56.1904859Z no-sudo: true 2025-12-04T08:56:56.1905131Z submodules: recursive 2025-12-04T08:56:56.1905412Z fetch-depth: 0 2025-12-04T08:56:56.1905664Z env: 2025-12-04T08:56:56.1905899Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:56:56.1906187Z ##[endgroup] 2025-12-04T08:56:56.1986544Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:56:56.1987823Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:56:56.1997825Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:56:56.1998278Z env: 2025-12-04T08:56:56.1998550Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:56:56.1998897Z ##[endgroup] 2025-12-04T08:56:56.2080119Z ##[group]Run # Use all available CPUs for fetching 2025-12-04T08:56:56.2080585Z # Use all available CPUs for fetching 2025-12-04T08:56:56.2080954Z cd "${GITHUB_WORKSPACE}" 2025-12-04T08:56:56.2081314Z git config --global fetch.parallel 0 2025-12-04T08:56:56.2081729Z git config --global submodule.fetchJobs 0 2025-12-04T08:56:56.2082079Z  2025-12-04T08:56:56.2082461Z # Clean workspace. The default checkout action should also do this, but 2025-12-04T08:56:56.2082963Z # do it here as well just in case 2025-12-04T08:56:56.2083303Z if [[ -d .git ]]; then 2025-12-04T08:56:56.2083603Z  if [ -z "${NO_SUDO}" ]; then 2025-12-04T08:56:56.2083935Z  sudo git clean -ffdx 2025-12-04T08:56:56.2084235Z  else 2025-12-04T08:56:56.2084472Z  git clean -ffdx 2025-12-04T08:56:56.2084754Z  fi 2025-12-04T08:56:56.2084982Z fi 2025-12-04T08:56:56.2090628Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:56:56.2091026Z env: 2025-12-04T08:56:56.2091340Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:56:56.2091637Z NO_SUDO: true 2025-12-04T08:56:56.2091858Z ##[endgroup] 2025-12-04T08:56:56.2235443Z ##[group]Run actions/checkout@v4 2025-12-04T08:56:56.2235797Z with: 2025-12-04T08:56:56.2236092Z ref: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:56:56.2236478Z fetch-depth: 0 2025-12-04T08:56:56.2236762Z submodules: recursive 2025-12-04T08:56:56.2237067Z show-progress: false 2025-12-04T08:56:56.2237367Z repository: pytorch/pytorch 2025-12-04T08:56:56.2237900Z token: *** 2025-12-04T08:56:56.2238163Z ssh-strict: true 2025-12-04T08:56:56.2238435Z ssh-user: git 2025-12-04T08:56:56.2238703Z persist-credentials: true 2025-12-04T08:56:56.2239019Z clean: true 2025-12-04T08:56:56.2239433Z sparse-checkout-cone-mode: true 2025-12-04T08:56:56.2239731Z fetch-tags: false 2025-12-04T08:56:56.2239977Z lfs: false 2025-12-04T08:56:56.2240215Z set-safe-directory: true 2025-12-04T08:56:56.2240484Z env: 2025-12-04T08:56:56.2240705Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:56:56.2240994Z ##[endgroup] 2025-12-04T08:56:56.3409325Z Syncing repository: pytorch/pytorch 2025-12-04T08:56:56.3410688Z ##[group]Getting Git version info 2025-12-04T08:56:56.3411202Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T08:56:56.3411895Z [command]/usr/bin/git version 2025-12-04T08:56:56.3412197Z git version 2.50.1 2025-12-04T08:56:56.3422119Z ##[endgroup] 2025-12-04T08:56:56.3432272Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/a6a80348-27fa-4934-b19c-c0b57eb963c5/.gitconfig' 2025-12-04T08:56:56.3451698Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a6a80348-27fa-4934-b19c-c0b57eb963c5' before making global git config changes 2025-12-04T08:56:56.3452822Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T08:56:56.3456838Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T08:56:56.3507105Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-12-04T08:56:56.3510920Z ##[group]Initializing the repository 2025-12-04T08:56:56.3515092Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T08:56:56.3547870Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-12-04T08:56:56.3548908Z hint: is subject to change. To configure the initial branch name to use in all 2025-12-04T08:56:56.3549602Z hint: of your new repositories, which will suppress this warning, call: 2025-12-04T08:56:56.3550094Z hint: 2025-12-04T08:56:56.3550419Z hint: git config --global init.defaultBranch 2025-12-04T08:56:56.3550834Z hint: 2025-12-04T08:56:56.3551225Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-12-04T08:56:56.3551916Z hint: 'development'. The just-created branch can be renamed via this command: 2025-12-04T08:56:56.3552426Z hint: 2025-12-04T08:56:56.3552697Z hint: git branch -m 2025-12-04T08:56:56.3553004Z hint: 2025-12-04T08:56:56.3553436Z hint: Disable this message with "git config set advice.defaultBranchName false" 2025-12-04T08:56:56.3554246Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-12-04T08:56:56.3556823Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-12-04T08:56:56.3583534Z ##[endgroup] 2025-12-04T08:56:56.3584046Z ##[group]Disabling automatic garbage collection 2025-12-04T08:56:56.3585703Z [command]/usr/bin/git config --local gc.auto 0 2025-12-04T08:56:56.3612777Z ##[endgroup] 2025-12-04T08:56:56.3613340Z ##[group]Setting up auth 2025-12-04T08:56:56.3618169Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T08:56:56.3647222Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T08:56:56.3970875Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T08:56:56.3996973Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T08:56:56.4283122Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T08:56:56.4310375Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T08:56:56.4595271Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T08:56:56.4641523Z ##[endgroup] 2025-12-04T08:56:56.4642152Z ##[group]Fetching the repository 2025-12-04T08:56:56.4647365Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-12-04T08:57:43.9839427Z From https://github.com/pytorch/pytorch 2025-12-04T08:57:43.9840024Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-12-04T08:57:43.9840683Z * [new branch] 2.9.1 -> origin/2.9.1 2025-12-04T08:57:43.9841347Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-12-04T08:57:43.9842071Z * [new branch] Flamefire-patch-1 -> origin/Flamefire-patch-1 2025-12-04T08:57:43.9842770Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-12-04T08:57:43.9843427Z * [new branch] HOPrintFunc -> origin/HOPrintFunc 2025-12-04T08:57:43.9844649Z * [new branch] IvanKobzarev/stack/1 -> origin/IvanKobzarev/stack/1 2025-12-04T08:57:43.9846812Z * [new branch] NicoshevSVE128 -> origin/NicoshevSVE128 2025-12-04T08:57:43.9847631Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-12-04T08:57:43.9848938Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-12-04T08:57:43.9850120Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-12-04T08:57:43.9851096Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-12-04T08:57:43.9852198Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-12-04T08:57:43.9853444Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-12-04T08:57:43.9854453Z * [new branch] VLA_exp -> origin/VLA_exp 2025-12-04T08:57:43.9856593Z * [new branch] activation_bench -> origin/activation_bench 2025-12-04T08:57:43.9857688Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-12-04T08:57:43.9859271Z * [new branch] adi/onednn_aarch64 -> origin/adi/onednn_aarch64 2025-12-04T08:57:43.9860351Z * [new branch] adi/test -> origin/adi/test 2025-12-04T08:57:43.9861473Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-12-04T08:57:43.9862607Z * [new branch] adi/test_m8g -> origin/adi/test_m8g 2025-12-04T08:57:43.9863723Z * [new branch] adi/test_onednn -> origin/adi/test_onednn 2025-12-04T08:57:43.9864869Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-12-04T08:57:43.9866035Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-12-04T08:57:43.9867044Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-12-04T08:57:43.9868977Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-12-04T08:57:43.9870961Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-12-04T08:57:43.9872189Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-12-04T08:57:43.9873550Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2025-12-04T08:57:43.9874568Z * [new branch] also-surround-shimh -> origin/also-surround-shimh 2025-12-04T08:57:43.9876308Z * [new branch] angelayi/aot_compile -> origin/angelayi/aot_compile 2025-12-04T08:57:43.9877476Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-12-04T08:57:43.9878533Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-12-04T08:57:43.9879788Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-12-04T08:57:43.9880873Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-12-04T08:57:43.9881974Z * [new branch] angelayi/inductor_const -> origin/angelayi/inductor_const 2025-12-04T08:57:43.9882967Z * [new branch] angelayi/lstm -> origin/angelayi/lstm 2025-12-04T08:57:43.9884533Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-12-04T08:57:43.9886002Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-12-04T08:57:43.9887109Z * [new branch] angelayi/side_eff -> origin/angelayi/side_eff 2025-12-04T08:57:43.9888296Z * [new branch] angelayi/state_dict -> origin/angelayi/state_dict 2025-12-04T08:57:43.9889571Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-12-04T08:57:43.9890816Z * [new branch] angelayi/symm_mem -> origin/angelayi/symm_mem 2025-12-04T08:57:43.9891838Z * [new branch] angelayi/test_cpp -> origin/angelayi/test_cpp 2025-12-04T08:57:43.9893002Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-12-04T08:57:43.9894068Z * [new branch] annotate_assert -> origin/annotate_assert 2025-12-04T08:57:43.9895265Z * [new branch] annotate_fallback_kernel -> origin/annotate_fallback_kernel 2025-12-04T08:57:43.9896370Z * [new branch] annotation_deepcopy -> origin/annotation_deepcopy 2025-12-04T08:57:43.9897467Z * [new branch] annotation_dynamo -> origin/annotation_dynamo 2025-12-04T08:57:43.9898615Z * [new branch] aot_eager_stack_trace -> origin/aot_eager_stack_trace 2025-12-04T08:57:43.9899711Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-12-04T08:57:43.9900854Z * [new branch] aoti_const_device -> origin/aoti_const_device 2025-12-04T08:57:43.9901961Z * [new branch] aoti_fqn_name_interface -> origin/aoti_fqn_name_interface 2025-12-04T08:57:43.9903079Z * [new branch] aoti_package_weights_binary -> origin/aoti_package_weights_binary 2025-12-04T08:57:43.9904117Z * [new branch] aoti_target_windows -> origin/aoti_target_windows 2025-12-04T08:57:43.9906176Z * [new branch] arsh/feat/inductor_check_profiling -> origin/arsh/feat/inductor_check_profiling 2025-12-04T08:57:43.9907148Z * [new branch] async_tp -> origin/async_tp 2025-12-04T08:57:43.9908893Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-12-04T08:57:43.9910047Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-12-04T08:57:43.9911202Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-12-04T08:57:43.9912447Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-12-04T08:57:43.9913656Z * [new branch] atalman-patch-4 -> origin/atalman-patch-4 2025-12-04T08:57:43.9914897Z * [new branch] atalman-patch-5 -> origin/atalman-patch-5 2025-12-04T08:57:43.9916107Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-12-04T08:57:43.9917328Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-12-04T08:57:43.9918559Z * [new branch] atalman-patch-8 -> origin/atalman-patch-8 2025-12-04T08:57:43.9920194Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-12-04T08:57:43.9921282Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-12-04T08:57:43.9922493Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-12-04T08:57:43.9923900Z * [new branch] attention_benchmarking_clean -> origin/attention_benchmarking_clean 2025-12-04T08:57:43.9925827Z * [new branch] bahuang/dt_fix_scalar_add -> origin/bahuang/dt_fix_scalar_add 2025-12-04T08:57:43.9926905Z * [new branch] bahuang/fix_debug_mode -> origin/bahuang/fix_debug_mode 2025-12-04T08:57:43.9928014Z * [new branch] bahuang/fix_expand -> origin/bahuang/fix_expand 2025-12-04T08:57:43.9929155Z * [new branch] bahuang/test -> origin/bahuang/test 2025-12-04T08:57:43.9930931Z * [new branch] base/1.5 -> origin/base/1.5 2025-12-04T08:57:43.9932423Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-12-04T08:57:43.9933407Z * [new branch] bench_scaled_mm_ops -> origin/bench_scaled_mm_ops 2025-12-04T08:57:43.9934719Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-12-04T08:57:43.9935748Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-12-04T08:57:43.9937455Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-12-04T08:57:43.9938963Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-12-04T08:57:43.9940514Z * [new branch] bf/bug-static-input -> origin/bf/bug-static-input 2025-12-04T08:57:43.9941509Z * [new branch] bf/cg-backend -> origin/bf/cg-backend 2025-12-04T08:57:43.9942559Z * [new branch] bf/cg-nccl-test -> origin/bf/cg-nccl-test 2025-12-04T08:57:43.9943621Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-12-04T08:57:43.9944792Z * [new branch] bf/clean-torchbench-hf -> origin/bf/clean-torchbench-hf 2025-12-04T08:57:43.9945847Z * [new branch] bf/combo-debug-log -> origin/bf/combo-debug-log 2025-12-04T08:57:43.9946889Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-12-04T08:57:43.9948999Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-12-04T08:57:43.9950450Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-12-04T08:57:43.9951503Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-12-04T08:57:43.9952448Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-12-04T08:57:43.9953631Z * [new branch] bf/dynamo-partition -> origin/bf/dynamo-partition 2025-12-04T08:57:43.9954730Z * [new branch] bf/lite -> origin/bf/lite 2025-12-04T08:57:43.9955932Z * [new branch] bf/pa-non-divisible -> origin/bf/pa-non-divisible 2025-12-04T08:57:43.9957221Z * [new branch] bf/partition-cache-free-symbols -> origin/bf/partition-cache-free-symbols 2025-12-04T08:57:43.9958356Z * [new branch] bf/partition-memory-plan -> origin/bf/partition-memory-plan 2025-12-04T08:57:43.9959492Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-12-04T08:57:43.9960836Z * [new branch] bf/partition-view-fallback -> origin/bf/partition-view-fallback 2025-12-04T08:57:43.9961889Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-12-04T08:57:43.9962922Z * [new branch] bf/timm-nov-26-2025 -> origin/bf/timm-nov-26-2025 2025-12-04T08:57:43.9964124Z * [new branch] bf/transformer-pin-4-57-3 -> origin/bf/transformer-pin-4-57-3 2025-12-04T08:57:43.9965274Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-12-04T08:57:43.9966325Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-12-04T08:57:43.9967391Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-12-04T08:57:43.9968441Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-12-04T08:57:43.9969522Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-12-04T08:57:43.9970630Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-12-04T08:57:43.9971732Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-12-04T08:57:43.9972739Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-12-04T08:57:43.9973841Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-12-04T08:57:43.9975098Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-12-04T08:57:43.9976056Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-12-04T08:57:43.9977093Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-12-04T08:57:43.9978213Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-12-04T08:57:43.9979281Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-12-04T08:57:43.9980539Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-12-04T08:57:43.9981584Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-12-04T08:57:43.9983167Z * [new branch] brister/fx_device_type -> origin/brister/fx_device_type 2025-12-04T08:57:43.9984240Z * [new branch] brister/test_inductor_all_fx -> origin/brister/test_inductor_all_fx 2025-12-04T08:57:43.9985427Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-12-04T08:57:43.9986405Z * [new branch] bwd-backup -> origin/bwd-backup 2025-12-04T08:57:43.9988117Z * [new branch] c57382a49 -> origin/c57382a49 2025-12-04T08:57:43.9989179Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-12-04T08:57:43.9990296Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-12-04T08:57:43.9992037Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-12-04T08:57:43.9993190Z * [new branch] cccclai-patch-1 -> origin/cccclai-patch-1 2025-12-04T08:57:43.9994581Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-12-04T08:57:43.9996324Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-12-04T08:57:43.9997526Z * [new branch] cherry-pick-162208-by-pytorch_bot_bot_ -> origin/cherry-pick-162208-by-pytorch_bot_bot_ 2025-12-04T08:57:43.9998721Z * [new branch] cherry-pick-163169-by-pytorch_bot_bot_ -> origin/cherry-pick-163169-by-pytorch_bot_bot_ 2025-12-04T08:57:43.9999984Z * [new branch] cherry-pick-165086-by-pytorch_bot_bot_ -> origin/cherry-pick-165086-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0001290Z * [new branch] cherry-pick-165514-by-pytorch_bot_bot_ -> origin/cherry-pick-165514-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0002464Z * [new branch] cherry-pick-165601-by-pytorch_bot_bot_ -> origin/cherry-pick-165601-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0003660Z * [new branch] cherry-pick-165667-by-pytorch_bot_bot_ -> origin/cherry-pick-165667-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0004966Z * [new branch] cherry-pick-165815-by-pytorch_bot_bot_ -> origin/cherry-pick-165815-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0006155Z * [new branch] cherry-pick-165922-by-pytorch_bot_bot_ -> origin/cherry-pick-165922-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0007301Z * [new branch] cherry-pick-166148-by-pytorch_bot_bot_ -> origin/cherry-pick-166148-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0008422Z * [new branch] cherry-pick-166181-by-pytorch_bot_bot_ -> origin/cherry-pick-166181-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0009578Z * [new branch] cherry-pick-166404-by-pytorch_bot_bot_ -> origin/cherry-pick-166404-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0010730Z * [new branch] cherry-pick-166427-by-pytorch_bot_bot_ -> origin/cherry-pick-166427-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0011945Z * [new branch] cherry-pick-166480-by-pytorch_bot_bot_ -> origin/cherry-pick-166480-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0012978Z * [new branch] cherry-pick-166570-by-pytorch_bot_bot_ -> origin/cherry-pick-166570-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0014152Z * [new branch] cherry-pick-166993-by-pytorch_bot_bot_ -> origin/cherry-pick-166993-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0015312Z * [new branch] cherry-pick-167111-by-pytorch_bot_bot_ -> origin/cherry-pick-167111-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0016482Z * [new branch] cherry-pick-167478-by-pytorch_bot_bot_ -> origin/cherry-pick-167478-by-pytorch_bot_bot_ 2025-12-04T08:57:44.0018017Z * [new branch] cherry_pick_166036_166040 -> origin/cherry_pick_166036_166040 2025-12-04T08:57:44.0019098Z * [new branch] cherry_pick_166457 -> origin/cherry_pick_166457 2025-12-04T08:57:44.0020321Z * [new branch] cherrypick_166338 -> origin/cherrypick_166338 2025-12-04T08:57:44.0021465Z * [new branch] cherrypick_166458 -> origin/cherrypick_166458 2025-12-04T08:57:44.0022587Z * [new branch] cherrypick_166586 -> origin/cherrypick_166586 2025-12-04T08:57:44.0023882Z * [new branch] cherrypick_166956 -> origin/cherrypick_166956 2025-12-04T08:57:44.0025427Z * [new branch] ci_attn -> origin/ci_attn 2025-12-04T08:57:44.0026461Z * [new branch] codex-testing -> origin/codex-testing 2025-12-04T08:57:44.0028648Z * [new branch] codex/add-check_memory_overlap-helper-functions -> origin/codex/add-check_memory_overlap-helper-functions 2025-12-04T08:57:44.0029662Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-12-04T08:57:44.0031241Z * [new branch] codex/investigate-segfaults-in-get_tensor_storage_id -> origin/codex/investigate-segfaults-in-get_tensor_storage_id 2025-12-04T08:57:44.0032576Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-12-04T08:57:44.0033505Z * [new branch] compatiblpy39util -> origin/compatiblpy39util 2025-12-04T08:57:44.0034576Z * [new branch] cond_hop_device -> origin/cond_hop_device 2025-12-04T08:57:44.0035702Z * [new branch] context_test -> origin/context_test 2025-12-04T08:57:44.0037566Z * [new branch] copilot/code-style-cleanup-python-pip -> origin/copilot/code-style-cleanup-python-pip 2025-12-04T08:57:44.0039035Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-12-04T08:57:44.0040193Z * [new branch] cpp-docs-dependency-upgrade -> origin/cpp-docs-dependency-upgrade 2025-12-04T08:57:44.0041691Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-12-04T08:57:44.0067609Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-12-04T08:57:44.0068649Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-12-04T08:57:44.0069306Z * [new branch] csl/clean_up -> origin/csl/clean_up 2025-12-04T08:57:44.0069959Z * [new branch] csl/fix_retry_segfault_exit -> origin/csl/fix_retry_segfault_exit 2025-12-04T08:57:44.0070595Z * [new branch] csl/katex -> origin/csl/katex 2025-12-04T08:57:44.0071154Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-12-04T08:57:44.0071765Z * [new branch] csl/lint_testing -> origin/csl/lint_testing 2025-12-04T08:57:44.0072358Z * [new branch] csl/lint_thing -> origin/csl/lint_thing 2025-12-04T08:57:44.0072984Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-12-04T08:57:44.0073825Z * [new branch] csl/manually_gen_json -> origin/csl/manually_gen_json 2025-12-04T08:57:44.0074560Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-12-04T08:57:44.0075200Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-12-04T08:57:44.0075829Z * [new branch] csl/print_timing -> origin/csl/print_timing 2025-12-04T08:57:44.0076467Z * [new branch] csl/remove_experiment -> origin/csl/remove_experiment 2025-12-04T08:57:44.0077162Z * [new branch] csl/remove_maybe_unused_var -> origin/csl/remove_maybe_unused_var 2025-12-04T08:57:44.0077957Z * [new branch] csl/remove_repo_specific_autolabel -> origin/csl/remove_repo_specific_autolabel 2025-12-04T08:57:44.0078737Z * [new branch] csl/remove_run_parallel -> origin/csl/remove_run_parallel 2025-12-04T08:57:44.0079474Z * [new branch] csl/remove_unused_vars -> origin/csl/remove_unused_vars 2025-12-04T08:57:44.0080164Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-12-04T08:57:44.0080875Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-12-04T08:57:44.0081498Z * [new branch] csl/smaller_avx_amx_runenrs -> origin/csl/smaller_avx_amx_runenrs 2025-12-04T08:57:44.0082133Z * [new branch] csl/td_job_level -> origin/csl/td_job_level 2025-12-04T08:57:44.0082808Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-12-04T08:57:44.0083627Z * [new branch] csl/test_owners_autograd_dispatch_nn -> origin/csl/test_owners_autograd_dispatch_nn 2025-12-04T08:57:44.0084440Z * [new branch] csl/test_owners_higher_confidence -> origin/csl/test_owners_higher_confidence 2025-12-04T08:57:44.0085170Z * [new branch] csl/upload_json_running -> origin/csl/upload_json_running 2025-12-04T08:57:44.0085791Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-12-04T08:57:44.0086397Z * [new branch] csl/xml_stuff -> origin/csl/xml_stuff 2025-12-04T08:57:44.0086956Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-12-04T08:57:44.0087508Z * [new branch] cuda_mempool -> origin/cuda_mempool 2025-12-04T08:57:44.0088080Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-12-04T08:57:44.0088734Z * [new branch] d4l3k/debug_plane_frtrace -> origin/d4l3k/debug_plane_frtrace 2025-12-04T08:57:44.0089354Z * [new branch] daxia6/2.8o3 -> origin/daxia6/2.8o3 2025-12-04T08:57:44.0089891Z * [new branch] debug-guard -> origin/debug-guard 2025-12-04T08:57:44.0090484Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-12-04T08:57:44.0091575Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.0 2025-12-04T08:57:44.0093108Z * [new branch] dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.57.1 2025-12-04T08:57:44.0094227Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-12-04T08:57:44.0095000Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-12-04T08:57:44.0095765Z * [new branch] dev/dhruva/flex_attn_opt -> origin/dev/dhruva/flex_attn_opt 2025-12-04T08:57:44.0096445Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-12-04T08:57:44.0097103Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-12-04T08:57:44.0097969Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-12-04T08:57:44.0098659Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-12-04T08:57:44.0099867Z * [new branch] dev/joona/fix_sdpa_memtest -> origin/dev/joona/fix_sdpa_memtest 2025-12-04T08:57:44.0101253Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-12-04T08:57:44.0102702Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-12-04T08:57:44.0104287Z * [new branch] dev/joona/scalar_clamp -> origin/dev/joona/scalar_clamp 2025-12-04T08:57:44.0105829Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-12-04T08:57:44.0107813Z * [new branch] dev/joona/sdpa_api -> origin/dev/joona/sdpa_api 2025-12-04T08:57:44.0109284Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-12-04T08:57:44.0110710Z * [new branch] dev/joona/ulpAssertClose -> origin/dev/joona/ulpAssertClose 2025-12-04T08:57:44.0111968Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-12-04T08:57:44.0113301Z * [new branch] disp_counter -> origin/disp_counter 2025-12-04T08:57:44.0114619Z * [new branch] divyanshk-patch-1 -> origin/divyanshk-patch-1 2025-12-04T08:57:44.0115616Z * [new branch] docs -> origin/docs 2025-12-04T08:57:44.0116813Z * [new branch] documentation -> origin/documentation 2025-12-04T08:57:44.0118033Z * [new branch] eager_model_benchmarks -> origin/eager_model_benchmarks 2025-12-04T08:57:44.0119896Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-12-04T08:57:44.0120886Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-12-04T08:57:44.0121860Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-12-04T08:57:44.0122992Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-12-04T08:57:44.0124687Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-12-04T08:57:44.0125858Z * [new branch] eqy-patch-3 -> origin/eqy-patch-3 2025-12-04T08:57:44.0127054Z * [new branch] eqy-patch-4 -> origin/eqy-patch-4 2025-12-04T08:57:44.0128339Z * [new branch] eqy-patch-5 -> origin/eqy-patch-5 2025-12-04T08:57:44.0129379Z * [new branch] eqy-patch-6 -> origin/eqy-patch-6 2025-12-04T08:57:44.0130964Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-12-04T08:57:44.0132170Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-12-04T08:57:44.0133202Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-12-04T08:57:44.0134410Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-12-04T08:57:44.0135594Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-12-04T08:57:44.0137071Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-12-04T08:57:44.0138531Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-12-04T08:57:44.0139495Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-12-04T08:57:44.0140756Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-12-04T08:57:44.0141999Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-12-04T08:57:44.0142912Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-12-04T08:57:44.0144283Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-12-04T08:57:44.0145175Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-12-04T08:57:44.0146304Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-12-04T08:57:44.0147870Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-12-04T08:57:44.0149020Z * [new branch] exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo 2025-12-04T08:57:44.0150190Z * [new branch] exclamaforte/profiler-visualization -> origin/exclamaforte/profiler-visualization 2025-12-04T08:57:44.0151396Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-12-04T08:57:44.0152592Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-12-04T08:57:44.0153743Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-12-04T08:57:44.0154699Z * [new branch] exec -> origin/exec 2025-12-04T08:57:44.0156185Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-12-04T08:57:44.0157319Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-12-04T08:57:44.0158518Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-12-04T08:57:44.0160005Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-12-04T08:57:44.0161020Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-12-04T08:57:44.0162107Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-12-04T08:57:44.0163225Z * [new branch] export-D80823877 -> origin/export-D80823877 2025-12-04T08:57:44.0164517Z * [new branch] export-D80958642 -> origin/export-D80958642 2025-12-04T08:57:44.0165565Z * [new branch] export-D81054193 -> origin/export-D81054193 2025-12-04T08:57:44.0166660Z * [new branch] export-D81204584 -> origin/export-D81204584 2025-12-04T08:57:44.0167810Z * [new branch] export-D81429090 -> origin/export-D81429090 2025-12-04T08:57:44.0169113Z * [new branch] export-D82250826 -> origin/export-D82250826 2025-12-04T08:57:44.0170157Z * [new branch] export-D82253817 -> origin/export-D82253817 2025-12-04T08:57:44.0171264Z * [new branch] export-D83541846 -> origin/export-D83541846 2025-12-04T08:57:44.0172400Z * [new branch] export-D83627170 -> origin/export-D83627170 2025-12-04T08:57:44.0173527Z * [new branch] export-D83766701 -> origin/export-D83766701 2025-12-04T08:57:44.0174732Z * [new branch] export-D83768878 -> origin/export-D83768878 2025-12-04T08:57:44.0175897Z * [new branch] export-D83769447 -> origin/export-D83769447 2025-12-04T08:57:44.0176944Z * [new branch] export-D84089824 -> origin/export-D84089824 2025-12-04T08:57:44.0178029Z * [new branch] export-D84213020 -> origin/export-D84213020 2025-12-04T08:57:44.0179717Z * [new branch] export-D84373821 -> origin/export-D84373821 2025-12-04T08:57:44.0180855Z * [new branch] export-D84612194 -> origin/export-D84612194 2025-12-04T08:57:44.0182146Z * [new branch] export-D84890985 -> origin/export-D84890985 2025-12-04T08:57:44.0183160Z * [new branch] export-D85122326 -> origin/export-D85122326 2025-12-04T08:57:44.0184464Z * [new branch] export-D86256198 -> origin/export-D86256198 2025-12-04T08:57:44.0185547Z * [new branch] export-D86460608 -> origin/export-D86460608 2025-12-04T08:57:44.0186882Z * [new branch] export-D86474796 -> origin/export-D86474796 2025-12-04T08:57:44.0188480Z * [new branch] export-D86712396 -> origin/export-D86712396 2025-12-04T08:57:44.0189628Z * [new branch] export-D87022129 -> origin/export-D87022129 2025-12-04T08:57:44.0190953Z * [new branch] export-D87838959 -> origin/export-D87838959 2025-12-04T08:57:44.0192116Z * [new branch] export-D88319437 -> origin/export-D88319437 2025-12-04T08:57:44.0193506Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-12-04T08:57:44.0194596Z * [new branch] ezyang-titan-october -> origin/ezyang-titan-october 2025-12-04T08:57:44.0196226Z * [new branch] ezyang-titan-october2 -> origin/ezyang-titan-october2 2025-12-04T08:57:44.0197263Z * [new branch] ezyang-war -> origin/ezyang-war 2025-12-04T08:57:44.0198950Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-12-04T08:57:44.0200064Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-12-04T08:57:44.0201754Z * [new branch] fadeputr/sequence_fbgemm -> origin/fadeputr/sequence_fbgemm 2025-12-04T08:57:44.0202823Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-12-04T08:57:44.0204454Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-12-04T08:57:44.0205681Z * [new branch] fca -> origin/fca 2025-12-04T08:57:44.0206729Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-12-04T08:57:44.0207791Z * [new branch] fca5 -> origin/fca5 2025-12-04T08:57:44.0209391Z * [new branch] feature/justknobs-cpp -> origin/feature/justknobs-cpp 2025-12-04T08:57:44.0210508Z * [new branch] feature/numa-forkserver -> origin/feature/numa-forkserver 2025-12-04T08:57:44.0212095Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-12-04T08:57:44.0213135Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-12-04T08:57:44.0214742Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-12-04T08:57:44.0215821Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-12-04T08:57:44.0216863Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-12-04T08:57:44.0217924Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-12-04T08:57:44.0218989Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-12-04T08:57:44.0220098Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-12-04T08:57:44.0221128Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-12-04T08:57:44.0222199Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-12-04T08:57:44.0224058Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-12-04T08:57:44.0225443Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-12-04T08:57:44.0226587Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-12-04T08:57:44.0228216Z * [new branch] fix_addmm_issue -> origin/fix_addmm_issue 2025-12-04T08:57:44.0229247Z * [new branch] fix_amd_missing_cluster_dims -> origin/fix_amd_missing_cluster_dims 2025-12-04T08:57:44.0230304Z * [new branch] fix_bench_bwd_pass -> origin/fix_bench_bwd_pass 2025-12-04T08:57:44.0231439Z * [new branch] fix_mem_profiler_config -> origin/fix_mem_profiler_config 2025-12-04T08:57:44.0232562Z * [new branch] fix_nvrtc_discovery -> origin/fix_nvrtc_discovery 2025-12-04T08:57:44.0233639Z * [new branch] fix_op_runner -> origin/fix_op_runner 2025-12-04T08:57:44.0234755Z * [new branch] fix_ubn_159469 -> origin/fix_ubn_159469 2025-12-04T08:57:44.0235965Z * [new branch] fixes-triage -> origin/fixes-triage 2025-12-04T08:57:44.0237103Z * [new branch] fixflashinfer -> origin/fixflashinfer 2025-12-04T08:57:44.0238455Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-12-04T08:57:44.0239571Z * [new branch] flex-flash -> origin/flex-flash 2025-12-04T08:57:44.0240822Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-12-04T08:57:44.0241826Z * [new branch] flex_flash -> origin/flex_flash 2025-12-04T08:57:44.0243519Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-12-04T08:57:44.0244586Z * [new branch] fmassa/tests_comm_compute_scheduler -> origin/fmassa/tests_comm_compute_scheduler 2025-12-04T08:57:44.0245613Z * [new branch] forkserver_fix -> origin/forkserver_fix 2025-12-04T08:57:44.0246755Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-12-04T08:57:44.0247839Z * [new branch] fx_cpp -> origin/fx_cpp 2025-12-04T08:57:44.0249523Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-12-04T08:57:44.0250819Z * [new branch] galv-patch-1 -> origin/galv-patch-1 2025-12-04T08:57:44.0252649Z * [new branch] galv/cudagraphs-conditional-nodes-4 -> origin/galv/cudagraphs-conditional-nodes-4 2025-12-04T08:57:44.0253998Z * [new branch] georgehong/cmakelists-patch -> origin/georgehong/cmakelists-patch 2025-12-04T08:57:44.0256347Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-12-04T08:57:44.0257408Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-12-04T08:57:44.0259358Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-12-04T08:57:44.0260470Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-12-04T08:57:44.0262758Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-12-04T08:57:44.0263823Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-12-04T08:57:44.0265708Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-12-04T08:57:44.0266800Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-12-04T08:57:44.0268272Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-12-04T08:57:44.0269906Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-12-04T08:57:44.0270947Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-12-04T08:57:44.0272086Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-12-04T08:57:44.0273736Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-12-04T08:57:44.0274870Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-12-04T08:57:44.0275895Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-12-04T08:57:44.0277420Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-12-04T08:57:44.0278459Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-12-04T08:57:44.0279728Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-12-04T08:57:44.0281238Z * [new branch] gh/H-Huang/226/base -> origin/gh/H-Huang/226/base 2025-12-04T08:57:44.0282666Z * [new branch] gh/H-Huang/226/head -> origin/gh/H-Huang/226/head 2025-12-04T08:57:44.0283746Z * [new branch] gh/H-Huang/226/orig -> origin/gh/H-Huang/226/orig 2025-12-04T08:57:44.0285273Z * [new branch] gh/H-Huang/228/base -> origin/gh/H-Huang/228/base 2025-12-04T08:57:44.0286314Z * [new branch] gh/H-Huang/228/head -> origin/gh/H-Huang/228/head 2025-12-04T08:57:44.0287401Z * [new branch] gh/H-Huang/228/orig -> origin/gh/H-Huang/228/orig 2025-12-04T08:57:44.0289354Z * [new branch] gh/IvanKobzarev/150/base -> origin/gh/IvanKobzarev/150/base 2025-12-04T08:57:44.0290397Z * [new branch] gh/IvanKobzarev/150/head -> origin/gh/IvanKobzarev/150/head 2025-12-04T08:57:44.0291489Z * [new branch] gh/IvanKobzarev/150/orig -> origin/gh/IvanKobzarev/150/orig 2025-12-04T08:57:44.0293169Z * [new branch] gh/IvanKobzarev/157/base -> origin/gh/IvanKobzarev/157/base 2025-12-04T08:57:44.0294258Z * [new branch] gh/IvanKobzarev/157/head -> origin/gh/IvanKobzarev/157/head 2025-12-04T08:57:44.0295409Z * [new branch] gh/IvanKobzarev/157/orig -> origin/gh/IvanKobzarev/157/orig 2025-12-04T08:57:44.0297065Z * [new branch] gh/IvanKobzarev/159/base -> origin/gh/IvanKobzarev/159/base 2025-12-04T08:57:44.0298112Z * [new branch] gh/IvanKobzarev/159/head -> origin/gh/IvanKobzarev/159/head 2025-12-04T08:57:44.0299221Z * [new branch] gh/IvanKobzarev/159/orig -> origin/gh/IvanKobzarev/159/orig 2025-12-04T08:57:44.0300817Z * [new branch] gh/IvanKobzarev/162/base -> origin/gh/IvanKobzarev/162/base 2025-12-04T08:57:44.0302076Z * [new branch] gh/IvanKobzarev/162/head -> origin/gh/IvanKobzarev/162/head 2025-12-04T08:57:44.0303134Z * [new branch] gh/IvanKobzarev/162/orig -> origin/gh/IvanKobzarev/162/orig 2025-12-04T08:57:44.0304748Z * [new branch] gh/IvanKobzarev/163/base -> origin/gh/IvanKobzarev/163/base 2025-12-04T08:57:44.0305880Z * [new branch] gh/IvanKobzarev/163/head -> origin/gh/IvanKobzarev/163/head 2025-12-04T08:57:44.0307000Z * [new branch] gh/IvanKobzarev/163/orig -> origin/gh/IvanKobzarev/163/orig 2025-12-04T08:57:44.0309031Z * [new branch] gh/IvanKobzarev/166/base -> origin/gh/IvanKobzarev/166/base 2025-12-04T08:57:44.0310090Z * [new branch] gh/IvanKobzarev/166/head -> origin/gh/IvanKobzarev/166/head 2025-12-04T08:57:44.0311241Z * [new branch] gh/IvanKobzarev/166/orig -> origin/gh/IvanKobzarev/166/orig 2025-12-04T08:57:44.0312903Z * [new branch] gh/IvanKobzarev/167/base -> origin/gh/IvanKobzarev/167/base 2025-12-04T08:57:44.0313921Z * [new branch] gh/IvanKobzarev/167/head -> origin/gh/IvanKobzarev/167/head 2025-12-04T08:57:44.0315007Z * [new branch] gh/IvanKobzarev/167/orig -> origin/gh/IvanKobzarev/167/orig 2025-12-04T08:57:44.0316580Z * [new branch] gh/IvanKobzarev/168/base -> origin/gh/IvanKobzarev/168/base 2025-12-04T08:57:44.0317660Z * [new branch] gh/IvanKobzarev/168/head -> origin/gh/IvanKobzarev/168/head 2025-12-04T08:57:44.0318910Z * [new branch] gh/IvanKobzarev/168/orig -> origin/gh/IvanKobzarev/168/orig 2025-12-04T08:57:44.0320539Z * [new branch] gh/IvanKobzarev/169/base -> origin/gh/IvanKobzarev/169/base 2025-12-04T08:57:44.0321595Z * [new branch] gh/IvanKobzarev/169/head -> origin/gh/IvanKobzarev/169/head 2025-12-04T08:57:44.0322694Z * [new branch] gh/IvanKobzarev/169/orig -> origin/gh/IvanKobzarev/169/orig 2025-12-04T08:57:44.0324530Z * [new branch] gh/IvanKobzarev/170/base -> origin/gh/IvanKobzarev/170/base 2025-12-04T08:57:44.0325651Z * [new branch] gh/IvanKobzarev/170/head -> origin/gh/IvanKobzarev/170/head 2025-12-04T08:57:44.0326828Z * [new branch] gh/IvanKobzarev/170/orig -> origin/gh/IvanKobzarev/170/orig 2025-12-04T08:57:44.0328635Z * [new branch] gh/IvanKobzarev/171/base -> origin/gh/IvanKobzarev/171/base 2025-12-04T08:57:44.0329688Z * [new branch] gh/IvanKobzarev/171/head -> origin/gh/IvanKobzarev/171/head 2025-12-04T08:57:44.0330836Z * [new branch] gh/IvanKobzarev/171/orig -> origin/gh/IvanKobzarev/171/orig 2025-12-04T08:57:44.0332497Z * [new branch] gh/IvanKobzarev/172/base -> origin/gh/IvanKobzarev/172/base 2025-12-04T08:57:44.0333649Z * [new branch] gh/IvanKobzarev/172/head -> origin/gh/IvanKobzarev/172/head 2025-12-04T08:57:44.0334819Z * [new branch] gh/IvanKobzarev/172/orig -> origin/gh/IvanKobzarev/172/orig 2025-12-04T08:57:44.0336627Z * [new branch] gh/IvanKobzarev/173/base -> origin/gh/IvanKobzarev/173/base 2025-12-04T08:57:44.0337677Z * [new branch] gh/IvanKobzarev/173/head -> origin/gh/IvanKobzarev/173/head 2025-12-04T08:57:44.0338788Z * [new branch] gh/IvanKobzarev/173/orig -> origin/gh/IvanKobzarev/173/orig 2025-12-04T08:57:44.0340407Z * [new branch] gh/IvanKobzarev/174/base -> origin/gh/IvanKobzarev/174/base 2025-12-04T08:57:44.0341577Z * [new branch] gh/IvanKobzarev/174/head -> origin/gh/IvanKobzarev/174/head 2025-12-04T08:57:44.0342687Z * [new branch] gh/IvanKobzarev/174/orig -> origin/gh/IvanKobzarev/174/orig 2025-12-04T08:57:44.0344254Z * [new branch] gh/IvanKobzarev/175/base -> origin/gh/IvanKobzarev/175/base 2025-12-04T08:57:44.0345437Z * [new branch] gh/IvanKobzarev/175/head -> origin/gh/IvanKobzarev/175/head 2025-12-04T08:57:44.0346509Z * [new branch] gh/IvanKobzarev/175/orig -> origin/gh/IvanKobzarev/175/orig 2025-12-04T08:57:44.0348651Z * [new branch] gh/IvanKobzarev/176/base -> origin/gh/IvanKobzarev/176/base 2025-12-04T08:57:44.0349728Z * [new branch] gh/IvanKobzarev/176/head -> origin/gh/IvanKobzarev/176/head 2025-12-04T08:57:44.0350893Z * [new branch] gh/IvanKobzarev/176/orig -> origin/gh/IvanKobzarev/176/orig 2025-12-04T08:57:44.0352767Z * [new branch] gh/IvanKobzarev/177/base -> origin/gh/IvanKobzarev/177/base 2025-12-04T08:57:44.0353891Z * [new branch] gh/IvanKobzarev/177/head -> origin/gh/IvanKobzarev/177/head 2025-12-04T08:57:44.0355075Z * [new branch] gh/IvanKobzarev/177/orig -> origin/gh/IvanKobzarev/177/orig 2025-12-04T08:57:44.0356782Z * [new branch] gh/IvanKobzarev/178/base -> origin/gh/IvanKobzarev/178/base 2025-12-04T08:57:44.0357931Z * [new branch] gh/IvanKobzarev/178/head -> origin/gh/IvanKobzarev/178/head 2025-12-04T08:57:44.0359087Z * [new branch] gh/IvanKobzarev/178/orig -> origin/gh/IvanKobzarev/178/orig 2025-12-04T08:57:44.0360853Z * [new branch] gh/IvanKobzarev/179/base -> origin/gh/IvanKobzarev/179/base 2025-12-04T08:57:44.0361824Z * [new branch] gh/IvanKobzarev/179/head -> origin/gh/IvanKobzarev/179/head 2025-12-04T08:57:44.0362936Z * [new branch] gh/IvanKobzarev/179/orig -> origin/gh/IvanKobzarev/179/orig 2025-12-04T08:57:44.0364753Z * [new branch] gh/IvanKobzarev/180/base -> origin/gh/IvanKobzarev/180/base 2025-12-04T08:57:44.0365765Z * [new branch] gh/IvanKobzarev/180/head -> origin/gh/IvanKobzarev/180/head 2025-12-04T08:57:44.0366918Z * [new branch] gh/IvanKobzarev/180/orig -> origin/gh/IvanKobzarev/180/orig 2025-12-04T08:57:44.0368708Z * [new branch] gh/IvanKobzarev/181/base -> origin/gh/IvanKobzarev/181/base 2025-12-04T08:57:44.0369826Z * [new branch] gh/IvanKobzarev/181/head -> origin/gh/IvanKobzarev/181/head 2025-12-04T08:57:44.0370970Z * [new branch] gh/IvanKobzarev/181/orig -> origin/gh/IvanKobzarev/181/orig 2025-12-04T08:57:44.0372765Z * [new branch] gh/IvanKobzarev/182/base -> origin/gh/IvanKobzarev/182/base 2025-12-04T08:57:44.0373954Z * [new branch] gh/IvanKobzarev/182/head -> origin/gh/IvanKobzarev/182/head 2025-12-04T08:57:44.0375062Z * [new branch] gh/IvanKobzarev/182/orig -> origin/gh/IvanKobzarev/182/orig 2025-12-04T08:57:44.0376795Z * [new branch] gh/IvanKobzarev/183/base -> origin/gh/IvanKobzarev/183/base 2025-12-04T08:57:44.0377931Z * [new branch] gh/IvanKobzarev/183/head -> origin/gh/IvanKobzarev/183/head 2025-12-04T08:57:44.0379058Z * [new branch] gh/IvanKobzarev/183/orig -> origin/gh/IvanKobzarev/183/orig 2025-12-04T08:57:44.0380655Z * [new branch] gh/IvanKobzarev/184/base -> origin/gh/IvanKobzarev/184/base 2025-12-04T08:57:44.0381730Z * [new branch] gh/IvanKobzarev/184/head -> origin/gh/IvanKobzarev/184/head 2025-12-04T08:57:44.0382877Z * [new branch] gh/IvanKobzarev/184/orig -> origin/gh/IvanKobzarev/184/orig 2025-12-04T08:57:44.0384752Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-12-04T08:57:44.0385949Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-12-04T08:57:44.0387403Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-12-04T08:57:44.0388714Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-12-04T08:57:44.0390565Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-12-04T08:57:44.0391724Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-12-04T08:57:44.0393308Z * [new branch] gh/NikhilAPatel/5/base -> origin/gh/NikhilAPatel/5/base 2025-12-04T08:57:44.0394386Z * [new branch] gh/NikhilAPatel/5/head -> origin/gh/NikhilAPatel/5/head 2025-12-04T08:57:44.0395564Z * [new branch] gh/NikhilAPatel/5/orig -> origin/gh/NikhilAPatel/5/orig 2025-12-04T08:57:44.0397402Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-12-04T08:57:44.0398492Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-12-04T08:57:44.0399651Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-12-04T08:57:44.0401364Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-12-04T08:57:44.0402410Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-12-04T08:57:44.0403515Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-12-04T08:57:44.0405090Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-12-04T08:57:44.0406208Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-12-04T08:57:44.0407262Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-12-04T08:57:44.0408746Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-12-04T08:57:44.0409784Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-12-04T08:57:44.0411003Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-12-04T08:57:44.0412298Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-12-04T08:57:44.0413368Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-12-04T08:57:44.0414454Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-12-04T08:57:44.0415996Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-12-04T08:57:44.0417020Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-12-04T08:57:44.0418078Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-12-04T08:57:44.0419552Z * [new branch] gh/PaliC/25/head -> origin/gh/PaliC/25/head 2025-12-04T08:57:44.0420600Z * [new branch] gh/PaliC/25/next -> origin/gh/PaliC/25/next 2025-12-04T08:57:44.0421745Z * [new branch] gh/PaliC/25/orig -> origin/gh/PaliC/25/orig 2025-12-04T08:57:44.0423307Z * [new branch] gh/PaliC/26/head -> origin/gh/PaliC/26/head 2025-12-04T08:57:44.0424597Z * [new branch] gh/PaliC/26/next -> origin/gh/PaliC/26/next 2025-12-04T08:57:44.0425785Z * [new branch] gh/PaliC/26/orig -> origin/gh/PaliC/26/orig 2025-12-04T08:57:44.0427457Z * [new branch] gh/PaliC/27/next -> origin/gh/PaliC/27/next 2025-12-04T08:57:44.0429006Z * [new branch] gh/PaliC/28/head -> origin/gh/PaliC/28/head 2025-12-04T08:57:44.0429924Z * [new branch] gh/PaliC/28/next -> origin/gh/PaliC/28/next 2025-12-04T08:57:44.0431055Z * [new branch] gh/PaliC/28/orig -> origin/gh/PaliC/28/orig 2025-12-04T08:57:44.0432643Z * [new branch] gh/PaliC/29/head -> origin/gh/PaliC/29/head 2025-12-04T08:57:44.0433579Z * [new branch] gh/PaliC/29/next -> origin/gh/PaliC/29/next 2025-12-04T08:57:44.0434683Z * [new branch] gh/PaliC/29/orig -> origin/gh/PaliC/29/orig 2025-12-04T08:57:44.0436242Z * [new branch] gh/PaliC/30/head -> origin/gh/PaliC/30/head 2025-12-04T08:57:44.0437189Z * [new branch] gh/PaliC/30/next -> origin/gh/PaliC/30/next 2025-12-04T08:57:44.0438361Z * [new branch] gh/PaliC/30/orig -> origin/gh/PaliC/30/orig 2025-12-04T08:57:44.0440103Z * [new branch] gh/PaliC/31/head -> origin/gh/PaliC/31/head 2025-12-04T08:57:44.0441059Z * [new branch] gh/PaliC/31/next -> origin/gh/PaliC/31/next 2025-12-04T08:57:44.0442149Z * [new branch] gh/PaliC/31/orig -> origin/gh/PaliC/31/orig 2025-12-04T08:57:44.0443955Z * [new branch] gh/PaulZhang12/25/base -> origin/gh/PaulZhang12/25/base 2025-12-04T08:57:44.0445126Z * [new branch] gh/PaulZhang12/25/head -> origin/gh/PaulZhang12/25/head 2025-12-04T08:57:44.0446261Z * [new branch] gh/PaulZhang12/25/orig -> origin/gh/PaulZhang12/25/orig 2025-12-04T08:57:44.0447828Z * [new branch] gh/PaulZhang12/28/base -> origin/gh/PaulZhang12/28/base 2025-12-04T08:57:44.0448954Z * [new branch] gh/PaulZhang12/28/head -> origin/gh/PaulZhang12/28/head 2025-12-04T08:57:44.0450054Z * [new branch] gh/PaulZhang12/28/orig -> origin/gh/PaulZhang12/28/orig 2025-12-04T08:57:44.0451816Z * [new branch] gh/PaulZhang12/31/base -> origin/gh/PaulZhang12/31/base 2025-12-04T08:57:44.0452832Z * [new branch] gh/PaulZhang12/31/head -> origin/gh/PaulZhang12/31/head 2025-12-04T08:57:44.0454040Z * [new branch] gh/PaulZhang12/31/orig -> origin/gh/PaulZhang12/31/orig 2025-12-04T08:57:44.0456286Z * [new branch] gh/PaulZhang12/37/base -> origin/gh/PaulZhang12/37/base 2025-12-04T08:57:44.0457574Z * [new branch] gh/PaulZhang12/37/head -> origin/gh/PaulZhang12/37/head 2025-12-04T08:57:44.0458327Z * [new branch] gh/PaulZhang12/37/orig -> origin/gh/PaulZhang12/37/orig 2025-12-04T08:57:44.0459301Z * [new branch] gh/PaulZhang12/40/base -> origin/gh/PaulZhang12/40/base 2025-12-04T08:57:44.0460372Z * [new branch] gh/PaulZhang12/40/head -> origin/gh/PaulZhang12/40/head 2025-12-04T08:57:44.0461470Z * [new branch] gh/PaulZhang12/40/orig -> origin/gh/PaulZhang12/40/orig 2025-12-04T08:57:44.0463072Z * [new branch] gh/PaulZhang12/42/base -> origin/gh/PaulZhang12/42/base 2025-12-04T08:57:44.0464121Z * [new branch] gh/PaulZhang12/42/head -> origin/gh/PaulZhang12/42/head 2025-12-04T08:57:44.0465706Z * [new branch] gh/PaulZhang12/43/base -> origin/gh/PaulZhang12/43/base 2025-12-04T08:57:44.0466801Z * [new branch] gh/PaulZhang12/43/head -> origin/gh/PaulZhang12/43/head 2025-12-04T08:57:44.0468228Z * [new branch] gh/PaulZhang12/43/orig -> origin/gh/PaulZhang12/43/orig 2025-12-04T08:57:44.0469674Z * [new branch] gh/PaulZhang12/44/base -> origin/gh/PaulZhang12/44/base 2025-12-04T08:57:44.0470767Z * [new branch] gh/PaulZhang12/44/head -> origin/gh/PaulZhang12/44/head 2025-12-04T08:57:44.0472457Z * [new branch] gh/PaulZhang12/45/base -> origin/gh/PaulZhang12/45/base 2025-12-04T08:57:44.0473472Z * [new branch] gh/PaulZhang12/45/head -> origin/gh/PaulZhang12/45/head 2025-12-04T08:57:44.0474593Z * [new branch] gh/PaulZhang12/45/orig -> origin/gh/PaulZhang12/45/orig 2025-12-04T08:57:44.0476210Z * [new branch] gh/PaulZhang12/46/base -> origin/gh/PaulZhang12/46/base 2025-12-04T08:57:44.0477291Z * [new branch] gh/PaulZhang12/46/head -> origin/gh/PaulZhang12/46/head 2025-12-04T08:57:44.0478455Z * [new branch] gh/PaulZhang12/46/orig -> origin/gh/PaulZhang12/46/orig 2025-12-04T08:57:44.0480132Z * [new branch] gh/PaulZhang12/47/base -> origin/gh/PaulZhang12/47/base 2025-12-04T08:57:44.0481283Z * [new branch] gh/PaulZhang12/47/head -> origin/gh/PaulZhang12/47/head 2025-12-04T08:57:44.0482421Z * [new branch] gh/PaulZhang12/47/orig -> origin/gh/PaulZhang12/47/orig 2025-12-04T08:57:44.0483826Z * [new branch] gh/PaulZhang12/48/base -> origin/gh/PaulZhang12/48/base 2025-12-04T08:57:44.0484888Z * [new branch] gh/PaulZhang12/48/head -> origin/gh/PaulZhang12/48/head 2025-12-04T08:57:44.0485963Z * [new branch] gh/PaulZhang12/48/orig -> origin/gh/PaulZhang12/48/orig 2025-12-04T08:57:44.0487740Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-12-04T08:57:44.0488796Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-12-04T08:57:44.0490835Z * [new branch] gh/SherlockNoMad/1/base -> origin/gh/SherlockNoMad/1/base 2025-12-04T08:57:44.0491878Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2025-12-04T08:57:44.0493487Z * [new branch] gh/SherlockNoMad/10/base -> origin/gh/SherlockNoMad/10/base 2025-12-04T08:57:44.0494581Z * [new branch] gh/SherlockNoMad/10/head -> origin/gh/SherlockNoMad/10/head 2025-12-04T08:57:44.0495791Z * [new branch] gh/SherlockNoMad/10/orig -> origin/gh/SherlockNoMad/10/orig 2025-12-04T08:57:44.0497206Z * [new branch] gh/SherlockNoMad/11/base -> origin/gh/SherlockNoMad/11/base 2025-12-04T08:57:44.0498219Z * [new branch] gh/SherlockNoMad/11/head -> origin/gh/SherlockNoMad/11/head 2025-12-04T08:57:44.0499349Z * [new branch] gh/SherlockNoMad/11/orig -> origin/gh/SherlockNoMad/11/orig 2025-12-04T08:57:44.0500867Z * [new branch] gh/SherlockNoMad/12/base -> origin/gh/SherlockNoMad/12/base 2025-12-04T08:57:44.0501788Z * [new branch] gh/SherlockNoMad/12/head -> origin/gh/SherlockNoMad/12/head 2025-12-04T08:57:44.0502860Z * [new branch] gh/SherlockNoMad/12/orig -> origin/gh/SherlockNoMad/12/orig 2025-12-04T08:57:44.0504433Z * [new branch] gh/SherlockNoMad/15/base -> origin/gh/SherlockNoMad/15/base 2025-12-04T08:57:44.0505709Z * [new branch] gh/SherlockNoMad/15/head -> origin/gh/SherlockNoMad/15/head 2025-12-04T08:57:44.0506830Z * [new branch] gh/SherlockNoMad/15/orig -> origin/gh/SherlockNoMad/15/orig 2025-12-04T08:57:44.0508828Z * [new branch] gh/SherlockNoMad/17/base -> origin/gh/SherlockNoMad/17/base 2025-12-04T08:57:44.0509799Z * [new branch] gh/SherlockNoMad/17/head -> origin/gh/SherlockNoMad/17/head 2025-12-04T08:57:44.0510948Z * [new branch] gh/SherlockNoMad/17/orig -> origin/gh/SherlockNoMad/17/orig 2025-12-04T08:57:44.0512679Z * [new branch] gh/SherlockNoMad/18/base -> origin/gh/SherlockNoMad/18/base 2025-12-04T08:57:44.0513817Z * [new branch] gh/SherlockNoMad/18/head -> origin/gh/SherlockNoMad/18/head 2025-12-04T08:57:44.0514991Z * [new branch] gh/SherlockNoMad/18/orig -> origin/gh/SherlockNoMad/18/orig 2025-12-04T08:57:44.0516430Z * [new branch] gh/SherlockNoMad/19/base -> origin/gh/SherlockNoMad/19/base 2025-12-04T08:57:44.0517550Z * [new branch] gh/SherlockNoMad/19/head -> origin/gh/SherlockNoMad/19/head 2025-12-04T08:57:44.0518808Z * [new branch] gh/SherlockNoMad/19/orig -> origin/gh/SherlockNoMad/19/orig 2025-12-04T08:57:44.0520316Z * [new branch] gh/SherlockNoMad/2/base -> origin/gh/SherlockNoMad/2/base 2025-12-04T08:57:44.0521294Z * [new branch] gh/SherlockNoMad/2/head -> origin/gh/SherlockNoMad/2/head 2025-12-04T08:57:44.0522696Z * [new branch] gh/SherlockNoMad/20/base -> origin/gh/SherlockNoMad/20/base 2025-12-04T08:57:44.0524028Z * [new branch] gh/SherlockNoMad/20/head -> origin/gh/SherlockNoMad/20/head 2025-12-04T08:57:44.0525352Z * [new branch] gh/SherlockNoMad/20/orig -> origin/gh/SherlockNoMad/20/orig 2025-12-04T08:57:44.0527171Z * [new branch] gh/SherlockNoMad/21/base -> origin/gh/SherlockNoMad/21/base 2025-12-04T08:57:44.0528344Z * [new branch] gh/SherlockNoMad/21/head -> origin/gh/SherlockNoMad/21/head 2025-12-04T08:57:44.0529407Z * [new branch] gh/SherlockNoMad/21/orig -> origin/gh/SherlockNoMad/21/orig 2025-12-04T08:57:44.0530829Z * [new branch] gh/SherlockNoMad/3/base -> origin/gh/SherlockNoMad/3/base 2025-12-04T08:57:44.0531828Z * [new branch] gh/SherlockNoMad/3/head -> origin/gh/SherlockNoMad/3/head 2025-12-04T08:57:44.0533264Z * [new branch] gh/SherlockNoMad/4/base -> origin/gh/SherlockNoMad/4/base 2025-12-04T08:57:44.0534236Z * [new branch] gh/SherlockNoMad/4/head -> origin/gh/SherlockNoMad/4/head 2025-12-04T08:57:44.0535665Z * [new branch] gh/SherlockNoMad/5/base -> origin/gh/SherlockNoMad/5/base 2025-12-04T08:57:44.0536775Z * [new branch] gh/SherlockNoMad/5/head -> origin/gh/SherlockNoMad/5/head 2025-12-04T08:57:44.0539152Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-12-04T08:57:44.0540476Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-12-04T08:57:44.0541789Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-12-04T08:57:44.0543432Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-12-04T08:57:44.0545215Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-12-04T08:57:44.0546417Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-12-04T08:57:44.0548164Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-12-04T08:57:44.0549279Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-12-04T08:57:44.0550773Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-12-04T08:57:44.0551753Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-12-04T08:57:44.0553306Z * [new branch] gh/StrongerXi/73/base -> origin/gh/StrongerXi/73/base 2025-12-04T08:57:44.0554377Z * [new branch] gh/StrongerXi/73/head -> origin/gh/StrongerXi/73/head 2025-12-04T08:57:44.0555531Z * [new branch] gh/StrongerXi/73/orig -> origin/gh/StrongerXi/73/orig 2025-12-04T08:57:44.0557536Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-12-04T08:57:44.0558552Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-12-04T08:57:44.0559728Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-12-04T08:57:44.0561358Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-12-04T08:57:44.0562434Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-12-04T08:57:44.0563525Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-12-04T08:57:44.0565177Z * [new branch] gh/XilunWu/168/base -> origin/gh/XilunWu/168/base 2025-12-04T08:57:44.0566173Z * [new branch] gh/XilunWu/168/head -> origin/gh/XilunWu/168/head 2025-12-04T08:57:44.0567259Z * [new branch] gh/XilunWu/168/orig -> origin/gh/XilunWu/168/orig 2025-12-04T08:57:44.0568841Z * [new branch] gh/XilunWu/169/base -> origin/gh/XilunWu/169/base 2025-12-04T08:57:44.0569863Z * [new branch] gh/XilunWu/169/head -> origin/gh/XilunWu/169/head 2025-12-04T08:57:44.0570956Z * [new branch] gh/XilunWu/169/orig -> origin/gh/XilunWu/169/orig 2025-12-04T08:57:44.0572325Z * [new branch] gh/XilunWu/170/base -> origin/gh/XilunWu/170/base 2025-12-04T08:57:44.0573492Z * [new branch] gh/XilunWu/170/head -> origin/gh/XilunWu/170/head 2025-12-04T08:57:44.0574595Z * [new branch] gh/XilunWu/170/orig -> origin/gh/XilunWu/170/orig 2025-12-04T08:57:44.0576242Z * [new branch] gh/XilunWu/171/base -> origin/gh/XilunWu/171/base 2025-12-04T08:57:44.0577320Z * [new branch] gh/XilunWu/171/head -> origin/gh/XilunWu/171/head 2025-12-04T08:57:44.0578433Z * [new branch] gh/XilunWu/171/orig -> origin/gh/XilunWu/171/orig 2025-12-04T08:57:44.0579921Z * [new branch] gh/XilunWu/173/base -> origin/gh/XilunWu/173/base 2025-12-04T08:57:44.0581015Z * [new branch] gh/XilunWu/173/head -> origin/gh/XilunWu/173/head 2025-12-04T08:57:44.0582134Z * [new branch] gh/XilunWu/173/orig -> origin/gh/XilunWu/173/orig 2025-12-04T08:57:44.0583653Z * [new branch] gh/XilunWu/175/base -> origin/gh/XilunWu/175/base 2025-12-04T08:57:44.0584715Z * [new branch] gh/XilunWu/175/head -> origin/gh/XilunWu/175/head 2025-12-04T08:57:44.0585829Z * [new branch] gh/XilunWu/175/orig -> origin/gh/XilunWu/175/orig 2025-12-04T08:57:44.0587510Z * [new branch] gh/XilunWu/176/base -> origin/gh/XilunWu/176/base 2025-12-04T08:57:44.0588885Z * [new branch] gh/XilunWu/176/head -> origin/gh/XilunWu/176/head 2025-12-04T08:57:44.0590108Z * [new branch] gh/XilunWu/176/orig -> origin/gh/XilunWu/176/orig 2025-12-04T08:57:44.0592104Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-12-04T08:57:44.0593204Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-12-04T08:57:44.0594280Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-12-04T08:57:44.0595875Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-12-04T08:57:44.0596978Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-12-04T08:57:44.0598212Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-12-04T08:57:44.0599728Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-12-04T08:57:44.0600883Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-12-04T08:57:44.0602107Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-12-04T08:57:44.0603637Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-12-04T08:57:44.0604708Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-12-04T08:57:44.0605819Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-12-04T08:57:44.0607360Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-12-04T08:57:44.0608399Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-12-04T08:57:44.0609521Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-12-04T08:57:44.0610967Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-12-04T08:57:44.0612031Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-12-04T08:57:44.0613145Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-12-04T08:57:44.0614648Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-12-04T08:57:44.0615687Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-12-04T08:57:44.0616845Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-12-04T08:57:44.0618306Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-12-04T08:57:44.0619359Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-12-04T08:57:44.0620465Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-12-04T08:57:44.0621976Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-12-04T08:57:44.0623052Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-12-04T08:57:44.0624421Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-12-04T08:57:44.0626248Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-12-04T08:57:44.0627336Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-12-04T08:57:44.0628566Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-12-04T08:57:44.0630132Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-12-04T08:57:44.0631218Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-12-04T08:57:44.0632363Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-12-04T08:57:44.0633940Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-12-04T08:57:44.0635065Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-12-04T08:57:44.0636376Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-12-04T08:57:44.0637998Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-12-04T08:57:44.0639155Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-12-04T08:57:44.0640679Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-12-04T08:57:44.0641717Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-12-04T08:57:44.0642822Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-12-04T08:57:44.0644360Z * [new branch] gh/XuehaiPan/390/base -> origin/gh/XuehaiPan/390/base 2025-12-04T08:57:44.0645423Z * [new branch] gh/XuehaiPan/390/head -> origin/gh/XuehaiPan/390/head 2025-12-04T08:57:44.0646556Z * [new branch] gh/XuehaiPan/390/orig -> origin/gh/XuehaiPan/390/orig 2025-12-04T08:57:44.0648082Z * [new branch] gh/XuehaiPan/391/base -> origin/gh/XuehaiPan/391/base 2025-12-04T08:57:44.0649122Z * [new branch] gh/XuehaiPan/391/head -> origin/gh/XuehaiPan/391/head 2025-12-04T08:57:44.0650197Z * [new branch] gh/XuehaiPan/391/orig -> origin/gh/XuehaiPan/391/orig 2025-12-04T08:57:44.0651710Z * [new branch] gh/XuehaiPan/392/base -> origin/gh/XuehaiPan/392/base 2025-12-04T08:57:44.0652836Z * [new branch] gh/XuehaiPan/392/head -> origin/gh/XuehaiPan/392/head 2025-12-04T08:57:44.0653894Z * [new branch] gh/XuehaiPan/392/orig -> origin/gh/XuehaiPan/392/orig 2025-12-04T08:57:44.0655880Z * [new branch] gh/XuehaiPan/394/base -> origin/gh/XuehaiPan/394/base 2025-12-04T08:57:44.0656922Z * [new branch] gh/XuehaiPan/394/head -> origin/gh/XuehaiPan/394/head 2025-12-04T08:57:44.0658028Z * [new branch] gh/XuehaiPan/394/orig -> origin/gh/XuehaiPan/394/orig 2025-12-04T08:57:44.0659606Z * [new branch] gh/XuehaiPan/397/base -> origin/gh/XuehaiPan/397/base 2025-12-04T08:57:44.0660659Z * [new branch] gh/XuehaiPan/397/head -> origin/gh/XuehaiPan/397/head 2025-12-04T08:57:44.0661769Z * [new branch] gh/XuehaiPan/397/orig -> origin/gh/XuehaiPan/397/orig 2025-12-04T08:57:44.0663334Z * [new branch] gh/XuehaiPan/398/base -> origin/gh/XuehaiPan/398/base 2025-12-04T08:57:44.0664402Z * [new branch] gh/XuehaiPan/398/head -> origin/gh/XuehaiPan/398/head 2025-12-04T08:57:44.0665487Z * [new branch] gh/XuehaiPan/398/orig -> origin/gh/XuehaiPan/398/orig 2025-12-04T08:57:44.0666992Z * [new branch] gh/XuehaiPan/399/base -> origin/gh/XuehaiPan/399/base 2025-12-04T08:57:44.0668375Z * [new branch] gh/XuehaiPan/399/head -> origin/gh/XuehaiPan/399/head 2025-12-04T08:57:44.0669542Z * [new branch] gh/XuehaiPan/399/orig -> origin/gh/XuehaiPan/399/orig 2025-12-04T08:57:44.0671160Z * [new branch] gh/XuehaiPan/400/base -> origin/gh/XuehaiPan/400/base 2025-12-04T08:57:44.0672232Z * [new branch] gh/XuehaiPan/400/head -> origin/gh/XuehaiPan/400/head 2025-12-04T08:57:44.0673398Z * [new branch] gh/XuehaiPan/400/orig -> origin/gh/XuehaiPan/400/orig 2025-12-04T08:57:44.0675295Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-12-04T08:57:44.0676350Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-12-04T08:57:44.0677550Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-12-04T08:57:44.0679096Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-12-04T08:57:44.0680374Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-12-04T08:57:44.0681785Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-12-04T08:57:44.0682731Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-12-04T08:57:44.0684346Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-12-04T08:57:44.0685399Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-12-04T08:57:44.0686988Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-12-04T08:57:44.0687986Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-12-04T08:57:44.0689720Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-12-04T08:57:44.0690563Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-12-04T08:57:44.0692028Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-12-04T08:57:44.0693022Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-12-04T08:57:44.0694449Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-12-04T08:57:44.0695436Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-12-04T08:57:44.0696510Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-12-04T08:57:44.0698417Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-12-04T08:57:44.0699537Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-12-04T08:57:44.0700912Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-12-04T08:57:44.0701900Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-12-04T08:57:44.0703638Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-12-04T08:57:44.0704702Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-12-04T08:57:44.0705785Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-12-04T08:57:44.0707633Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2025-12-04T08:57:44.0708954Z * [new branch] gh/albanD/4/head -> origin/gh/albanD/4/head 2025-12-04T08:57:44.0710151Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2025-12-04T08:57:44.0712068Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-12-04T08:57:44.0713590Z * [new branch] gh/alexsamardzic/12/base -> origin/gh/alexsamardzic/12/base 2025-12-04T08:57:44.0714780Z * [new branch] gh/alexsamardzic/12/head -> origin/gh/alexsamardzic/12/head 2025-12-04T08:57:44.0715927Z * [new branch] gh/alexsamardzic/12/orig -> origin/gh/alexsamardzic/12/orig 2025-12-04T08:57:44.0717528Z * [new branch] gh/alexsamardzic/14/base -> origin/gh/alexsamardzic/14/base 2025-12-04T08:57:44.0718642Z * [new branch] gh/alexsamardzic/14/head -> origin/gh/alexsamardzic/14/head 2025-12-04T08:57:44.0719790Z * [new branch] gh/alexsamardzic/14/orig -> origin/gh/alexsamardzic/14/orig 2025-12-04T08:57:44.0721536Z * [new branch] gh/alexsamardzic/15/base -> origin/gh/alexsamardzic/15/base 2025-12-04T08:57:44.0722593Z * [new branch] gh/alexsamardzic/15/head -> origin/gh/alexsamardzic/15/head 2025-12-04T08:57:44.0723840Z * [new branch] gh/alexsamardzic/15/orig -> origin/gh/alexsamardzic/15/orig 2025-12-04T08:57:44.0726156Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-12-04T08:57:44.0727119Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-12-04T08:57:44.0728258Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-12-04T08:57:44.0730253Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-12-04T08:57:44.0731499Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-12-04T08:57:44.0732690Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-12-04T08:57:44.0734516Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-12-04T08:57:44.0735681Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-12-04T08:57:44.0737043Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-12-04T08:57:44.0738911Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-12-04T08:57:44.0740177Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-12-04T08:57:44.0741785Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-12-04T08:57:44.0743110Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-12-04T08:57:44.0744601Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-12-04T08:57:44.0745731Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-12-04T08:57:44.0746856Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-12-04T08:57:44.0749003Z * [new branch] gh/andyanwang/42/base -> origin/gh/andyanwang/42/base 2025-12-04T08:57:44.0750064Z * [new branch] gh/andyanwang/42/head -> origin/gh/andyanwang/42/head 2025-12-04T08:57:44.0751872Z * [new branch] gh/andyanwang/42/orig -> origin/gh/andyanwang/42/orig 2025-12-04T08:57:44.0753558Z * [new branch] gh/andyanwang/45/base -> origin/gh/andyanwang/45/base 2025-12-04T08:57:44.0754724Z * [new branch] gh/andyanwang/45/head -> origin/gh/andyanwang/45/head 2025-12-04T08:57:44.0755909Z * [new branch] gh/andyanwang/45/orig -> origin/gh/andyanwang/45/orig 2025-12-04T08:57:44.0757791Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-12-04T08:57:44.0758889Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-12-04T08:57:44.0760628Z * [new branch] gh/angelayi/114/base -> origin/gh/angelayi/114/base 2025-12-04T08:57:44.0761730Z * [new branch] gh/angelayi/114/head -> origin/gh/angelayi/114/head 2025-12-04T08:57:44.0763318Z * [new branch] gh/angelayi/114/orig -> origin/gh/angelayi/114/orig 2025-12-04T08:57:44.0764816Z * [new branch] gh/angelayi/116/base -> origin/gh/angelayi/116/base 2025-12-04T08:57:44.0765865Z * [new branch] gh/angelayi/116/head -> origin/gh/angelayi/116/head 2025-12-04T08:57:44.0766972Z * [new branch] gh/angelayi/116/orig -> origin/gh/angelayi/116/orig 2025-12-04T08:57:44.0768682Z * [new branch] gh/angelayi/122/base -> origin/gh/angelayi/122/base 2025-12-04T08:57:44.0769646Z * [new branch] gh/angelayi/122/head -> origin/gh/angelayi/122/head 2025-12-04T08:57:44.0770770Z * [new branch] gh/angelayi/122/orig -> origin/gh/angelayi/122/orig 2025-12-04T08:57:44.0772432Z * [new branch] gh/angelayi/124/base -> origin/gh/angelayi/124/base 2025-12-04T08:57:44.0773440Z * [new branch] gh/angelayi/124/head -> origin/gh/angelayi/124/head 2025-12-04T08:57:44.0774877Z * [new branch] gh/angelayi/124/orig -> origin/gh/angelayi/124/orig 2025-12-04T08:57:44.0776445Z * [new branch] gh/angelayi/128/base -> origin/gh/angelayi/128/base 2025-12-04T08:57:44.0777558Z * [new branch] gh/angelayi/128/head -> origin/gh/angelayi/128/head 2025-12-04T08:57:44.0778624Z * [new branch] gh/angelayi/128/orig -> origin/gh/angelayi/128/orig 2025-12-04T08:57:44.0780186Z * [new branch] gh/angelayi/131/base -> origin/gh/angelayi/131/base 2025-12-04T08:57:44.0781263Z * [new branch] gh/angelayi/131/head -> origin/gh/angelayi/131/head 2025-12-04T08:57:44.0782382Z * [new branch] gh/angelayi/131/orig -> origin/gh/angelayi/131/orig 2025-12-04T08:57:44.0784177Z * [new branch] gh/angelayi/132/base -> origin/gh/angelayi/132/base 2025-12-04T08:57:44.0785419Z * [new branch] gh/angelayi/132/head -> origin/gh/angelayi/132/head 2025-12-04T08:57:44.0786768Z * [new branch] gh/angelayi/132/orig -> origin/gh/angelayi/132/orig 2025-12-04T08:57:44.0788551Z * [new branch] gh/angelayi/133/base -> origin/gh/angelayi/133/base 2025-12-04T08:57:44.0789683Z * [new branch] gh/angelayi/133/head -> origin/gh/angelayi/133/head 2025-12-04T08:57:44.0790833Z * [new branch] gh/angelayi/133/orig -> origin/gh/angelayi/133/orig 2025-12-04T08:57:44.0792709Z * [new branch] gh/angelayi/134/base -> origin/gh/angelayi/134/base 2025-12-04T08:57:44.0794080Z * [new branch] gh/angelayi/134/head -> origin/gh/angelayi/134/head 2025-12-04T08:57:44.0795155Z * [new branch] gh/angelayi/134/orig -> origin/gh/angelayi/134/orig 2025-12-04T08:57:44.0796879Z * [new branch] gh/angelayi/135/base -> origin/gh/angelayi/135/base 2025-12-04T08:57:44.0798100Z * [new branch] gh/angelayi/135/head -> origin/gh/angelayi/135/head 2025-12-04T08:57:44.0799254Z * [new branch] gh/angelayi/135/orig -> origin/gh/angelayi/135/orig 2025-12-04T08:57:44.0800884Z * [new branch] gh/angelayi/136/base -> origin/gh/angelayi/136/base 2025-12-04T08:57:44.0801916Z * [new branch] gh/angelayi/136/head -> origin/gh/angelayi/136/head 2025-12-04T08:57:44.0803060Z * [new branch] gh/angelayi/136/orig -> origin/gh/angelayi/136/orig 2025-12-04T08:57:44.0804611Z * [new branch] gh/angelayi/137/base -> origin/gh/angelayi/137/base 2025-12-04T08:57:44.0805604Z * [new branch] gh/angelayi/137/head -> origin/gh/angelayi/137/head 2025-12-04T08:57:44.0806989Z * [new branch] gh/angelayi/137/orig -> origin/gh/angelayi/137/orig 2025-12-04T08:57:44.0808421Z * [new branch] gh/angelayi/138/base -> origin/gh/angelayi/138/base 2025-12-04T08:57:44.0809414Z * [new branch] gh/angelayi/138/head -> origin/gh/angelayi/138/head 2025-12-04T08:57:44.0810506Z * [new branch] gh/angelayi/138/orig -> origin/gh/angelayi/138/orig 2025-12-04T08:57:44.0812032Z * [new branch] gh/angelayi/139/base -> origin/gh/angelayi/139/base 2025-12-04T08:57:44.0813084Z * [new branch] gh/angelayi/139/head -> origin/gh/angelayi/139/head 2025-12-04T08:57:44.0814194Z * [new branch] gh/angelayi/139/orig -> origin/gh/angelayi/139/orig 2025-12-04T08:57:44.0815834Z * [new branch] gh/angelayi/140/base -> origin/gh/angelayi/140/base 2025-12-04T08:57:44.0816950Z * [new branch] gh/angelayi/140/head -> origin/gh/angelayi/140/head 2025-12-04T08:57:44.0818113Z * [new branch] gh/angelayi/140/orig -> origin/gh/angelayi/140/orig 2025-12-04T08:57:44.0820044Z * [new branch] gh/angelayi/141/base -> origin/gh/angelayi/141/base 2025-12-04T08:57:44.0821212Z * [new branch] gh/angelayi/141/head -> origin/gh/angelayi/141/head 2025-12-04T08:57:44.0822258Z * [new branch] gh/angelayi/141/orig -> origin/gh/angelayi/141/orig 2025-12-04T08:57:44.0824180Z * [new branch] gh/angelayi/142/base -> origin/gh/angelayi/142/base 2025-12-04T08:57:44.0826772Z * [new branch] gh/angelayi/142/head -> origin/gh/angelayi/142/head 2025-12-04T08:57:44.0828059Z * [new branch] gh/angelayi/142/orig -> origin/gh/angelayi/142/orig 2025-12-04T08:57:44.0829804Z * [new branch] gh/angelayi/143/base -> origin/gh/angelayi/143/base 2025-12-04T08:57:44.0830863Z * [new branch] gh/angelayi/143/head -> origin/gh/angelayi/143/head 2025-12-04T08:57:44.0831992Z * [new branch] gh/angelayi/143/orig -> origin/gh/angelayi/143/orig 2025-12-04T08:57:44.0833626Z * [new branch] gh/angelayi/144/base -> origin/gh/angelayi/144/base 2025-12-04T08:57:44.0834827Z * [new branch] gh/angelayi/144/head -> origin/gh/angelayi/144/head 2025-12-04T08:57:44.0836041Z * [new branch] gh/angelayi/144/orig -> origin/gh/angelayi/144/orig 2025-12-04T08:57:44.0838143Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-12-04T08:57:44.0839381Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-12-04T08:57:44.0840469Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-12-04T08:57:44.0842154Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-12-04T08:57:44.0843185Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-12-04T08:57:44.0844322Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-12-04T08:57:44.0845915Z * [new branch] gh/anijain2305/854/base -> origin/gh/anijain2305/854/base 2025-12-04T08:57:44.0847057Z * [new branch] gh/anijain2305/854/head -> origin/gh/anijain2305/854/head 2025-12-04T08:57:44.0848145Z * [new branch] gh/anijain2305/854/orig -> origin/gh/anijain2305/854/orig 2025-12-04T08:57:44.0849790Z * [new branch] gh/anijain2305/864/base -> origin/gh/anijain2305/864/base 2025-12-04T08:57:44.0850833Z * [new branch] gh/anijain2305/864/head -> origin/gh/anijain2305/864/head 2025-12-04T08:57:44.0851965Z * [new branch] gh/anijain2305/864/orig -> origin/gh/anijain2305/864/orig 2025-12-04T08:57:44.0853586Z * [new branch] gh/anijain2305/870/base -> origin/gh/anijain2305/870/base 2025-12-04T08:57:44.0854581Z * [new branch] gh/anijain2305/870/head -> origin/gh/anijain2305/870/head 2025-12-04T08:57:44.0855712Z * [new branch] gh/anijain2305/870/orig -> origin/gh/anijain2305/870/orig 2025-12-04T08:57:44.0857380Z * [new branch] gh/anijain2305/873/base -> origin/gh/anijain2305/873/base 2025-12-04T08:57:44.0858345Z * [new branch] gh/anijain2305/873/head -> origin/gh/anijain2305/873/head 2025-12-04T08:57:44.0859411Z * [new branch] gh/anijain2305/873/orig -> origin/gh/anijain2305/873/orig 2025-12-04T08:57:44.0860933Z * [new branch] gh/anijain2305/894/base -> origin/gh/anijain2305/894/base 2025-12-04T08:57:44.0861975Z * [new branch] gh/anijain2305/894/head -> origin/gh/anijain2305/894/head 2025-12-04T08:57:44.0863111Z * [new branch] gh/anijain2305/894/orig -> origin/gh/anijain2305/894/orig 2025-12-04T08:57:44.0864710Z * [new branch] gh/anijain2305/895/base -> origin/gh/anijain2305/895/base 2025-12-04T08:57:44.0865833Z * [new branch] gh/anijain2305/895/head -> origin/gh/anijain2305/895/head 2025-12-04T08:57:44.0866933Z * [new branch] gh/anijain2305/895/orig -> origin/gh/anijain2305/895/orig 2025-12-04T08:57:44.0869019Z * [new branch] gh/anijain2305/910/base -> origin/gh/anijain2305/910/base 2025-12-04T08:57:44.0869987Z * [new branch] gh/anijain2305/910/head -> origin/gh/anijain2305/910/head 2025-12-04T08:57:44.0871124Z * [new branch] gh/anijain2305/910/orig -> origin/gh/anijain2305/910/orig 2025-12-04T08:57:44.0872797Z * [new branch] gh/anijain2305/919/base -> origin/gh/anijain2305/919/base 2025-12-04T08:57:44.0873965Z * [new branch] gh/anijain2305/919/head -> origin/gh/anijain2305/919/head 2025-12-04T08:57:44.0875145Z * [new branch] gh/anijain2305/919/orig -> origin/gh/anijain2305/919/orig 2025-12-04T08:57:44.0876770Z * [new branch] gh/anijain2305/922/base -> origin/gh/anijain2305/922/base 2025-12-04T08:57:44.0877976Z * [new branch] gh/anijain2305/922/head -> origin/gh/anijain2305/922/head 2025-12-04T08:57:44.0879419Z * [new branch] gh/anijain2305/922/orig -> origin/gh/anijain2305/922/orig 2025-12-04T08:57:44.0881123Z * [new branch] gh/anijain2305/932/base -> origin/gh/anijain2305/932/base 2025-12-04T08:57:44.0882310Z * [new branch] gh/anijain2305/932/head -> origin/gh/anijain2305/932/head 2025-12-04T08:57:44.0883479Z * [new branch] gh/anijain2305/932/orig -> origin/gh/anijain2305/932/orig 2025-12-04T08:57:44.0885017Z * [new branch] gh/anijain2305/940/base -> origin/gh/anijain2305/940/base 2025-12-04T08:57:44.0886073Z * [new branch] gh/anijain2305/940/head -> origin/gh/anijain2305/940/head 2025-12-04T08:57:44.0887179Z * [new branch] gh/anijain2305/940/orig -> origin/gh/anijain2305/940/orig 2025-12-04T08:57:44.0888720Z * [new branch] gh/anijain2305/941/base -> origin/gh/anijain2305/941/base 2025-12-04T08:57:44.0889764Z * [new branch] gh/anijain2305/941/head -> origin/gh/anijain2305/941/head 2025-12-04T08:57:44.0890885Z * [new branch] gh/anijain2305/941/orig -> origin/gh/anijain2305/941/orig 2025-12-04T08:57:44.0892395Z * [new branch] gh/anijain2305/942/base -> origin/gh/anijain2305/942/base 2025-12-04T08:57:44.0893569Z * [new branch] gh/anijain2305/942/head -> origin/gh/anijain2305/942/head 2025-12-04T08:57:44.0894772Z * [new branch] gh/anijain2305/942/orig -> origin/gh/anijain2305/942/orig 2025-12-04T08:57:44.0896363Z * [new branch] gh/anijain2305/943/base -> origin/gh/anijain2305/943/base 2025-12-04T08:57:44.0897400Z * [new branch] gh/anijain2305/943/head -> origin/gh/anijain2305/943/head 2025-12-04T08:57:44.0898502Z * [new branch] gh/anijain2305/943/orig -> origin/gh/anijain2305/943/orig 2025-12-04T08:57:44.0900562Z * [new branch] gh/anijain2305/944/base -> origin/gh/anijain2305/944/base 2025-12-04T08:57:44.0901581Z * [new branch] gh/anijain2305/944/head -> origin/gh/anijain2305/944/head 2025-12-04T08:57:44.0902681Z * [new branch] gh/anijain2305/944/orig -> origin/gh/anijain2305/944/orig 2025-12-04T08:57:44.0904996Z * [new branch] gh/anijain2305/945/base -> origin/gh/anijain2305/945/base 2025-12-04T08:57:44.0906113Z * [new branch] gh/anijain2305/945/head -> origin/gh/anijain2305/945/head 2025-12-04T08:57:44.0907306Z * [new branch] gh/anijain2305/945/orig -> origin/gh/anijain2305/945/orig 2025-12-04T08:57:44.0909312Z * [new branch] gh/anijain2305/946/base -> origin/gh/anijain2305/946/base 2025-12-04T08:57:44.0910383Z * [new branch] gh/anijain2305/946/head -> origin/gh/anijain2305/946/head 2025-12-04T08:57:44.0911537Z * [new branch] gh/anijain2305/946/orig -> origin/gh/anijain2305/946/orig 2025-12-04T08:57:44.0913119Z * [new branch] gh/anijain2305/947/base -> origin/gh/anijain2305/947/base 2025-12-04T08:57:44.0914293Z * [new branch] gh/anijain2305/947/head -> origin/gh/anijain2305/947/head 2025-12-04T08:57:44.0915357Z * [new branch] gh/anijain2305/947/orig -> origin/gh/anijain2305/947/orig 2025-12-04T08:57:44.0917230Z * [new branch] gh/anijain2305/948/base -> origin/gh/anijain2305/948/base 2025-12-04T08:57:44.0918058Z * [new branch] gh/anijain2305/948/head -> origin/gh/anijain2305/948/head 2025-12-04T08:57:44.0919202Z * [new branch] gh/anijain2305/948/orig -> origin/gh/anijain2305/948/orig 2025-12-04T08:57:44.0920923Z * [new branch] gh/anijain2305/949/base -> origin/gh/anijain2305/949/base 2025-12-04T08:57:44.0921979Z * [new branch] gh/anijain2305/949/head -> origin/gh/anijain2305/949/head 2025-12-04T08:57:44.0923092Z * [new branch] gh/anijain2305/949/orig -> origin/gh/anijain2305/949/orig 2025-12-04T08:57:44.0925209Z * [new branch] gh/anijain2305/950/base -> origin/gh/anijain2305/950/base 2025-12-04T08:57:44.0926349Z * [new branch] gh/anijain2305/950/head -> origin/gh/anijain2305/950/head 2025-12-04T08:57:44.0927487Z * [new branch] gh/anijain2305/950/orig -> origin/gh/anijain2305/950/orig 2025-12-04T08:57:44.0929195Z * [new branch] gh/anijain2305/951/base -> origin/gh/anijain2305/951/base 2025-12-04T08:57:44.0930233Z * [new branch] gh/anijain2305/951/head -> origin/gh/anijain2305/951/head 2025-12-04T08:57:44.0931374Z * [new branch] gh/anijain2305/951/orig -> origin/gh/anijain2305/951/orig 2025-12-04T08:57:44.0933047Z * [new branch] gh/anijain2305/952/base -> origin/gh/anijain2305/952/base 2025-12-04T08:57:44.0934112Z * [new branch] gh/anijain2305/952/head -> origin/gh/anijain2305/952/head 2025-12-04T08:57:44.0935233Z * [new branch] gh/anijain2305/952/orig -> origin/gh/anijain2305/952/orig 2025-12-04T08:57:44.0936930Z * [new branch] gh/anijain2305/953/base -> origin/gh/anijain2305/953/base 2025-12-04T08:57:44.0937947Z * [new branch] gh/anijain2305/953/head -> origin/gh/anijain2305/953/head 2025-12-04T08:57:44.0939087Z * [new branch] gh/anijain2305/953/orig -> origin/gh/anijain2305/953/orig 2025-12-04T08:57:44.0940642Z * [new branch] gh/anijain2305/954/base -> origin/gh/anijain2305/954/base 2025-12-04T08:57:44.0941750Z * [new branch] gh/anijain2305/954/head -> origin/gh/anijain2305/954/head 2025-12-04T08:57:44.0942854Z * [new branch] gh/anijain2305/954/orig -> origin/gh/anijain2305/954/orig 2025-12-04T08:57:44.0944765Z * [new branch] gh/anijain2305/955/base -> origin/gh/anijain2305/955/base 2025-12-04T08:57:44.0945589Z * [new branch] gh/anijain2305/955/head -> origin/gh/anijain2305/955/head 2025-12-04T08:57:44.0946856Z * [new branch] gh/anijain2305/955/orig -> origin/gh/anijain2305/955/orig 2025-12-04T08:57:44.0948868Z * [new branch] gh/anijain2305/956/base -> origin/gh/anijain2305/956/base 2025-12-04T08:57:44.0949959Z * [new branch] gh/anijain2305/956/head -> origin/gh/anijain2305/956/head 2025-12-04T08:57:44.0951113Z * [new branch] gh/anijain2305/956/orig -> origin/gh/anijain2305/956/orig 2025-12-04T08:57:44.0952786Z * [new branch] gh/anijain2305/957/base -> origin/gh/anijain2305/957/base 2025-12-04T08:57:44.0953887Z * [new branch] gh/anijain2305/957/head -> origin/gh/anijain2305/957/head 2025-12-04T08:57:44.0955065Z * [new branch] gh/anijain2305/957/orig -> origin/gh/anijain2305/957/orig 2025-12-04T08:57:44.0956759Z * [new branch] gh/anijain2305/958/base -> origin/gh/anijain2305/958/base 2025-12-04T08:57:44.0957924Z * [new branch] gh/anijain2305/958/head -> origin/gh/anijain2305/958/head 2025-12-04T08:57:44.0959152Z * [new branch] gh/anijain2305/958/orig -> origin/gh/anijain2305/958/orig 2025-12-04T08:57:44.0960741Z * [new branch] gh/anijain2305/959/base -> origin/gh/anijain2305/959/base 2025-12-04T08:57:44.0961865Z * [new branch] gh/anijain2305/959/head -> origin/gh/anijain2305/959/head 2025-12-04T08:57:44.0963015Z * [new branch] gh/anijain2305/959/orig -> origin/gh/anijain2305/959/orig 2025-12-04T08:57:44.0964638Z * [new branch] gh/anijain2305/960/base -> origin/gh/anijain2305/960/base 2025-12-04T08:57:44.0965720Z * [new branch] gh/anijain2305/960/head -> origin/gh/anijain2305/960/head 2025-12-04T08:57:44.0966819Z * [new branch] gh/anijain2305/960/orig -> origin/gh/anijain2305/960/orig 2025-12-04T08:57:44.0968478Z * [new branch] gh/anijain2305/961/base -> origin/gh/anijain2305/961/base 2025-12-04T08:57:44.0969540Z * [new branch] gh/anijain2305/961/head -> origin/gh/anijain2305/961/head 2025-12-04T08:57:44.0970673Z * [new branch] gh/anijain2305/961/orig -> origin/gh/anijain2305/961/orig 2025-12-04T08:57:44.0972313Z * [new branch] gh/anijain2305/962/base -> origin/gh/anijain2305/962/base 2025-12-04T08:57:44.0973353Z * [new branch] gh/anijain2305/962/head -> origin/gh/anijain2305/962/head 2025-12-04T08:57:44.0974458Z * [new branch] gh/anijain2305/962/orig -> origin/gh/anijain2305/962/orig 2025-12-04T08:57:44.0976425Z * [new branch] gh/anijain2305/963/base -> origin/gh/anijain2305/963/base 2025-12-04T08:57:44.0977663Z * [new branch] gh/anijain2305/963/head -> origin/gh/anijain2305/963/head 2025-12-04T08:57:44.0978814Z * [new branch] gh/anijain2305/963/orig -> origin/gh/anijain2305/963/orig 2025-12-04T08:57:44.0980448Z * [new branch] gh/anijain2305/964/base -> origin/gh/anijain2305/964/base 2025-12-04T08:57:44.0982095Z * [new branch] gh/anijain2305/964/head -> origin/gh/anijain2305/964/head 2025-12-04T08:57:44.0983161Z * [new branch] gh/anijain2305/964/orig -> origin/gh/anijain2305/964/orig 2025-12-04T08:57:44.0984734Z * [new branch] gh/anijain2305/965/base -> origin/gh/anijain2305/965/base 2025-12-04T08:57:44.0985827Z * [new branch] gh/anijain2305/965/head -> origin/gh/anijain2305/965/head 2025-12-04T08:57:44.0986996Z * [new branch] gh/anijain2305/965/orig -> origin/gh/anijain2305/965/orig 2025-12-04T08:57:44.0988780Z * [new branch] gh/anijain2305/966/base -> origin/gh/anijain2305/966/base 2025-12-04T08:57:44.0989858Z * [new branch] gh/anijain2305/966/head -> origin/gh/anijain2305/966/head 2025-12-04T08:57:44.0990998Z * [new branch] gh/anijain2305/966/orig -> origin/gh/anijain2305/966/orig 2025-12-04T08:57:44.0992667Z * [new branch] gh/anijain2305/967/base -> origin/gh/anijain2305/967/base 2025-12-04T08:57:44.0993736Z * [new branch] gh/anijain2305/967/head -> origin/gh/anijain2305/967/head 2025-12-04T08:57:44.0995447Z * [new branch] gh/anijain2305/967/orig -> origin/gh/anijain2305/967/orig 2025-12-04T08:57:44.0997427Z * [new branch] gh/anijain2305/968/base -> origin/gh/anijain2305/968/base 2025-12-04T08:57:44.0998547Z * [new branch] gh/anijain2305/968/head -> origin/gh/anijain2305/968/head 2025-12-04T08:57:44.0999663Z * [new branch] gh/anijain2305/968/orig -> origin/gh/anijain2305/968/orig 2025-12-04T08:57:44.1001494Z * [new branch] gh/anijain2305/969/base -> origin/gh/anijain2305/969/base 2025-12-04T08:57:44.1002611Z * [new branch] gh/anijain2305/969/head -> origin/gh/anijain2305/969/head 2025-12-04T08:57:44.1003841Z * [new branch] gh/anijain2305/969/orig -> origin/gh/anijain2305/969/orig 2025-12-04T08:57:44.1005616Z * [new branch] gh/anijain2305/970/base -> origin/gh/anijain2305/970/base 2025-12-04T08:57:44.1006656Z * [new branch] gh/anijain2305/970/head -> origin/gh/anijain2305/970/head 2025-12-04T08:57:44.1007806Z * [new branch] gh/anijain2305/970/orig -> origin/gh/anijain2305/970/orig 2025-12-04T08:57:44.1009701Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-12-04T08:57:44.1010754Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-12-04T08:57:44.1011825Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-12-04T08:57:44.1013806Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-12-04T08:57:44.1014911Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-12-04T08:57:44.1016441Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-12-04T08:57:44.1017505Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-12-04T08:57:44.1018857Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-12-04T08:57:44.1019868Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-12-04T08:57:44.1021561Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-12-04T08:57:44.1022253Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-12-04T08:57:44.1023771Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-12-04T08:57:44.1025054Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-12-04T08:57:44.1026882Z * [new branch] gh/anshul-si/53/base -> origin/gh/anshul-si/53/base 2025-12-04T08:57:44.1028066Z * [new branch] gh/anshul-si/53/head -> origin/gh/anshul-si/53/head 2025-12-04T08:57:44.1029716Z * [new branch] gh/anshul-si/58/base -> origin/gh/anshul-si/58/base 2025-12-04T08:57:44.1030842Z * [new branch] gh/anshul-si/58/head -> origin/gh/anshul-si/58/head 2025-12-04T08:57:44.1032279Z * [new branch] gh/anshul-si/66/base -> origin/gh/anshul-si/66/base 2025-12-04T08:57:44.1033367Z * [new branch] gh/anshul-si/66/head -> origin/gh/anshul-si/66/head 2025-12-04T08:57:44.1034496Z * [new branch] gh/anshul-si/66/orig -> origin/gh/anshul-si/66/orig 2025-12-04T08:57:44.1035925Z * [new branch] gh/anshul-si/67/base -> origin/gh/anshul-si/67/base 2025-12-04T08:57:44.1037033Z * [new branch] gh/anshul-si/67/head -> origin/gh/anshul-si/67/head 2025-12-04T08:57:44.1038195Z * [new branch] gh/anshul-si/67/orig -> origin/gh/anshul-si/67/orig 2025-12-04T08:57:44.1040021Z * [new branch] gh/anshul-si/68/base -> origin/gh/anshul-si/68/base 2025-12-04T08:57:44.1041047Z * [new branch] gh/anshul-si/68/head -> origin/gh/anshul-si/68/head 2025-12-04T08:57:44.1042113Z * [new branch] gh/anshul-si/68/orig -> origin/gh/anshul-si/68/orig 2025-12-04T08:57:44.1043908Z * [new branch] gh/anshul-si/69/base -> origin/gh/anshul-si/69/base 2025-12-04T08:57:44.1044978Z * [new branch] gh/anshul-si/69/head -> origin/gh/anshul-si/69/head 2025-12-04T08:57:44.1046097Z * [new branch] gh/anshul-si/69/orig -> origin/gh/anshul-si/69/orig 2025-12-04T08:57:44.1047657Z * [new branch] gh/anshul-si/70/base -> origin/gh/anshul-si/70/base 2025-12-04T08:57:44.1048815Z * [new branch] gh/anshul-si/70/head -> origin/gh/anshul-si/70/head 2025-12-04T08:57:44.1049987Z * [new branch] gh/anshul-si/70/orig -> origin/gh/anshul-si/70/orig 2025-12-04T08:57:44.1051641Z * [new branch] gh/anshul-si/71/base -> origin/gh/anshul-si/71/base 2025-12-04T08:57:44.1052646Z * [new branch] gh/anshul-si/71/head -> origin/gh/anshul-si/71/head 2025-12-04T08:57:44.1053752Z * [new branch] gh/anshul-si/71/orig -> origin/gh/anshul-si/71/orig 2025-12-04T08:57:44.1055383Z * [new branch] gh/anshul-si/72/base -> origin/gh/anshul-si/72/base 2025-12-04T08:57:44.1056466Z * [new branch] gh/anshul-si/72/head -> origin/gh/anshul-si/72/head 2025-12-04T08:57:44.1057555Z * [new branch] gh/anshul-si/72/orig -> origin/gh/anshul-si/72/orig 2025-12-04T08:57:44.1059119Z * [new branch] gh/anshul-si/73/base -> origin/gh/anshul-si/73/base 2025-12-04T08:57:44.1060247Z * [new branch] gh/anshul-si/73/head -> origin/gh/anshul-si/73/head 2025-12-04T08:57:44.1061415Z * [new branch] gh/anshul-si/73/orig -> origin/gh/anshul-si/73/orig 2025-12-04T08:57:44.1063378Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-12-04T08:57:44.1064441Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-12-04T08:57:44.1066187Z * [new branch] gh/aorenste/134/base -> origin/gh/aorenste/134/base 2025-12-04T08:57:44.1067446Z * [new branch] gh/aorenste/134/head -> origin/gh/aorenste/134/head 2025-12-04T08:57:44.1068820Z * [new branch] gh/aorenste/134/orig -> origin/gh/aorenste/134/orig 2025-12-04T08:57:44.1070525Z * [new branch] gh/aorenste/139/base -> origin/gh/aorenste/139/base 2025-12-04T08:57:44.1071637Z * [new branch] gh/aorenste/139/head -> origin/gh/aorenste/139/head 2025-12-04T08:57:44.1072820Z * [new branch] gh/aorenste/139/orig -> origin/gh/aorenste/139/orig 2025-12-04T08:57:44.1074497Z * [new branch] gh/aorenste/141/base -> origin/gh/aorenste/141/base 2025-12-04T08:57:44.1075586Z * [new branch] gh/aorenste/141/head -> origin/gh/aorenste/141/head 2025-12-04T08:57:44.1077531Z * [new branch] gh/aorenste/145/base -> origin/gh/aorenste/145/base 2025-12-04T08:57:44.1078569Z * [new branch] gh/aorenste/145/head -> origin/gh/aorenste/145/head 2025-12-04T08:57:44.1080061Z * [new branch] gh/aorenste/145/orig -> origin/gh/aorenste/145/orig 2025-12-04T08:57:44.1081686Z * [new branch] gh/aorenste/146/base -> origin/gh/aorenste/146/base 2025-12-04T08:57:44.1083063Z * [new branch] gh/aorenste/146/head -> origin/gh/aorenste/146/head 2025-12-04T08:57:44.1084190Z * [new branch] gh/aorenste/146/orig -> origin/gh/aorenste/146/orig 2025-12-04T08:57:44.1085834Z * [new branch] gh/aorenste/147/base -> origin/gh/aorenste/147/base 2025-12-04T08:57:44.1087054Z * [new branch] gh/aorenste/147/head -> origin/gh/aorenste/147/head 2025-12-04T08:57:44.1088180Z * [new branch] gh/aorenste/147/orig -> origin/gh/aorenste/147/orig 2025-12-04T08:57:44.1089849Z * [new branch] gh/aorenste/148/base -> origin/gh/aorenste/148/base 2025-12-04T08:57:44.1090949Z * [new branch] gh/aorenste/148/head -> origin/gh/aorenste/148/head 2025-12-04T08:57:44.1092123Z * [new branch] gh/aorenste/148/orig -> origin/gh/aorenste/148/orig 2025-12-04T08:57:44.1093658Z * [new branch] gh/aorenste/149/base -> origin/gh/aorenste/149/base 2025-12-04T08:57:44.1094696Z * [new branch] gh/aorenste/149/head -> origin/gh/aorenste/149/head 2025-12-04T08:57:44.1095807Z * [new branch] gh/aorenste/149/orig -> origin/gh/aorenste/149/orig 2025-12-04T08:57:44.1097324Z * [new branch] gh/aorenste/150/base -> origin/gh/aorenste/150/base 2025-12-04T08:57:44.1098460Z * [new branch] gh/aorenste/150/head -> origin/gh/aorenste/150/head 2025-12-04T08:57:44.1099549Z * [new branch] gh/aorenste/150/orig -> origin/gh/aorenste/150/orig 2025-12-04T08:57:44.1101030Z * [new branch] gh/aorenste/151/base -> origin/gh/aorenste/151/base 2025-12-04T08:57:44.1102072Z * [new branch] gh/aorenste/151/head -> origin/gh/aorenste/151/head 2025-12-04T08:57:44.1103207Z * [new branch] gh/aorenste/151/orig -> origin/gh/aorenste/151/orig 2025-12-04T08:57:44.1104886Z * [new branch] gh/aorenste/152/base -> origin/gh/aorenste/152/base 2025-12-04T08:57:44.1105954Z * [new branch] gh/aorenste/152/head -> origin/gh/aorenste/152/head 2025-12-04T08:57:44.1107125Z * [new branch] gh/aorenste/152/orig -> origin/gh/aorenste/152/orig 2025-12-04T08:57:44.1128873Z * [new branch] gh/aorenste/153/base -> origin/gh/aorenste/153/base 2025-12-04T08:57:44.1129820Z * [new branch] gh/aorenste/153/head -> origin/gh/aorenste/153/head 2025-12-04T08:57:44.1130469Z * [new branch] gh/aorenste/153/orig -> origin/gh/aorenste/153/orig 2025-12-04T08:57:44.1131113Z * [new branch] gh/aorenste/154/base -> origin/gh/aorenste/154/base 2025-12-04T08:57:44.1131752Z * [new branch] gh/aorenste/154/head -> origin/gh/aorenste/154/head 2025-12-04T08:57:44.1132379Z * [new branch] gh/aorenste/154/orig -> origin/gh/aorenste/154/orig 2025-12-04T08:57:44.1133013Z * [new branch] gh/aorenste/155/base -> origin/gh/aorenste/155/base 2025-12-04T08:57:44.1133650Z * [new branch] gh/aorenste/155/head -> origin/gh/aorenste/155/head 2025-12-04T08:57:44.1134290Z * [new branch] gh/aorenste/155/orig -> origin/gh/aorenste/155/orig 2025-12-04T08:57:44.1134911Z * [new branch] gh/aorenste/156/base -> origin/gh/aorenste/156/base 2025-12-04T08:57:44.1135553Z * [new branch] gh/aorenste/156/head -> origin/gh/aorenste/156/head 2025-12-04T08:57:44.1136194Z * [new branch] gh/aorenste/156/orig -> origin/gh/aorenste/156/orig 2025-12-04T08:57:44.1136930Z * [new branch] gh/aorenste/157/base -> origin/gh/aorenste/157/base 2025-12-04T08:57:44.1137534Z * [new branch] gh/aorenste/157/head -> origin/gh/aorenste/157/head 2025-12-04T08:57:44.1138151Z * [new branch] gh/aorenste/157/orig -> origin/gh/aorenste/157/orig 2025-12-04T08:57:44.1138767Z * [new branch] gh/aorenste/158/base -> origin/gh/aorenste/158/base 2025-12-04T08:57:44.1139383Z * [new branch] gh/aorenste/158/head -> origin/gh/aorenste/158/head 2025-12-04T08:57:44.1139995Z * [new branch] gh/aorenste/158/orig -> origin/gh/aorenste/158/orig 2025-12-04T08:57:44.1140622Z * [new branch] gh/aorenste/159/base -> origin/gh/aorenste/159/base 2025-12-04T08:57:44.1141236Z * [new branch] gh/aorenste/159/head -> origin/gh/aorenste/159/head 2025-12-04T08:57:44.1141845Z * [new branch] gh/aorenste/159/orig -> origin/gh/aorenste/159/orig 2025-12-04T08:57:44.1142503Z * [new branch] gh/avikchaudhuri/1/base -> origin/gh/avikchaudhuri/1/base 2025-12-04T08:57:44.1143177Z * [new branch] gh/avikchaudhuri/1/head -> origin/gh/avikchaudhuri/1/head 2025-12-04T08:57:44.1143840Z * [new branch] gh/avikchaudhuri/2/base -> origin/gh/avikchaudhuri/2/base 2025-12-04T08:57:44.1144501Z * [new branch] gh/avikchaudhuri/2/head -> origin/gh/avikchaudhuri/2/head 2025-12-04T08:57:44.1145168Z * [new branch] gh/avikchaudhuri/2/orig -> origin/gh/avikchaudhuri/2/orig 2025-12-04T08:57:44.1145810Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-12-04T08:57:44.1146585Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-12-04T08:57:44.1147374Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-12-04T08:57:44.1148173Z * [new branch] gh/bdhirsh/668/base -> origin/gh/bdhirsh/668/base 2025-12-04T08:57:44.1148802Z * [new branch] gh/bdhirsh/668/head -> origin/gh/bdhirsh/668/head 2025-12-04T08:57:44.1149884Z * [new branch] gh/bdhirsh/668/orig -> origin/gh/bdhirsh/668/orig 2025-12-04T08:57:44.1151726Z * [new branch] gh/bdhirsh/669/base -> origin/gh/bdhirsh/669/base 2025-12-04T08:57:44.1152739Z * [new branch] gh/bdhirsh/669/head -> origin/gh/bdhirsh/669/head 2025-12-04T08:57:44.1153843Z * [new branch] gh/bdhirsh/669/orig -> origin/gh/bdhirsh/669/orig 2025-12-04T08:57:44.1156109Z * [new branch] gh/bdhirsh/670/base -> origin/gh/bdhirsh/670/base 2025-12-04T08:57:44.1157355Z * [new branch] gh/bdhirsh/670/head -> origin/gh/bdhirsh/670/head 2025-12-04T08:57:44.1158452Z * [new branch] gh/bdhirsh/670/orig -> origin/gh/bdhirsh/670/orig 2025-12-04T08:57:44.1160191Z * [new branch] gh/bdhirsh/672/base -> origin/gh/bdhirsh/672/base 2025-12-04T08:57:44.1161264Z * [new branch] gh/bdhirsh/672/head -> origin/gh/bdhirsh/672/head 2025-12-04T08:57:44.1162347Z * [new branch] gh/bdhirsh/672/orig -> origin/gh/bdhirsh/672/orig 2025-12-04T08:57:44.1164195Z * [new branch] gh/bdhirsh/675/base -> origin/gh/bdhirsh/675/base 2025-12-04T08:57:44.1165396Z * [new branch] gh/bdhirsh/675/head -> origin/gh/bdhirsh/675/head 2025-12-04T08:57:44.1166501Z * [new branch] gh/bdhirsh/675/orig -> origin/gh/bdhirsh/675/orig 2025-12-04T08:57:44.1168651Z * [new branch] gh/bdhirsh/676/base -> origin/gh/bdhirsh/676/base 2025-12-04T08:57:44.1169853Z * [new branch] gh/bdhirsh/676/head -> origin/gh/bdhirsh/676/head 2025-12-04T08:57:44.1170963Z * [new branch] gh/bdhirsh/676/orig -> origin/gh/bdhirsh/676/orig 2025-12-04T08:57:44.1172512Z * [new branch] gh/bdhirsh/677/base -> origin/gh/bdhirsh/677/base 2025-12-04T08:57:44.1174033Z * [new branch] gh/bdhirsh/677/head -> origin/gh/bdhirsh/677/head 2025-12-04T08:57:44.1175170Z * [new branch] gh/bdhirsh/677/orig -> origin/gh/bdhirsh/677/orig 2025-12-04T08:57:44.1176802Z * [new branch] gh/bdhirsh/678/base -> origin/gh/bdhirsh/678/base 2025-12-04T08:57:44.1177995Z * [new branch] gh/bdhirsh/678/head -> origin/gh/bdhirsh/678/head 2025-12-04T08:57:44.1179132Z * [new branch] gh/bdhirsh/678/orig -> origin/gh/bdhirsh/678/orig 2025-12-04T08:57:44.1180853Z * [new branch] gh/bdhirsh/679/base -> origin/gh/bdhirsh/679/base 2025-12-04T08:57:44.1182024Z * [new branch] gh/bdhirsh/679/head -> origin/gh/bdhirsh/679/head 2025-12-04T08:57:44.1183143Z * [new branch] gh/bdhirsh/679/orig -> origin/gh/bdhirsh/679/orig 2025-12-04T08:57:44.1184745Z * [new branch] gh/bdhirsh/680/base -> origin/gh/bdhirsh/680/base 2025-12-04T08:57:44.1185930Z * [new branch] gh/bdhirsh/680/head -> origin/gh/bdhirsh/680/head 2025-12-04T08:57:44.1187078Z * [new branch] gh/bdhirsh/680/orig -> origin/gh/bdhirsh/680/orig 2025-12-04T08:57:44.1188753Z * [new branch] gh/bdhirsh/681/base -> origin/gh/bdhirsh/681/base 2025-12-04T08:57:44.1189981Z * [new branch] gh/bdhirsh/681/head -> origin/gh/bdhirsh/681/head 2025-12-04T08:57:44.1191193Z * [new branch] gh/bdhirsh/681/orig -> origin/gh/bdhirsh/681/orig 2025-12-04T08:57:44.1193159Z * [new branch] gh/benjaminglass1/101/base -> origin/gh/benjaminglass1/101/base 2025-12-04T08:57:44.1194258Z * [new branch] gh/benjaminglass1/101/head -> origin/gh/benjaminglass1/101/head 2025-12-04T08:57:44.1195438Z * [new branch] gh/benjaminglass1/101/orig -> origin/gh/benjaminglass1/101/orig 2025-12-04T08:57:44.1196947Z * [new branch] gh/benjaminglass1/102/base -> origin/gh/benjaminglass1/102/base 2025-12-04T08:57:44.1198086Z * [new branch] gh/benjaminglass1/102/head -> origin/gh/benjaminglass1/102/head 2025-12-04T08:57:44.1199322Z * [new branch] gh/benjaminglass1/102/orig -> origin/gh/benjaminglass1/102/orig 2025-12-04T08:57:44.1200944Z * [new branch] gh/benjaminglass1/106/base -> origin/gh/benjaminglass1/106/base 2025-12-04T08:57:44.1202031Z * [new branch] gh/benjaminglass1/106/head -> origin/gh/benjaminglass1/106/head 2025-12-04T08:57:44.1203144Z * [new branch] gh/benjaminglass1/106/orig -> origin/gh/benjaminglass1/106/orig 2025-12-04T08:57:44.1204598Z * [new branch] gh/benjaminglass1/107/base -> origin/gh/benjaminglass1/107/base 2025-12-04T08:57:44.1205686Z * [new branch] gh/benjaminglass1/107/head -> origin/gh/benjaminglass1/107/head 2025-12-04T08:57:44.1206821Z * [new branch] gh/benjaminglass1/107/orig -> origin/gh/benjaminglass1/107/orig 2025-12-04T08:57:44.1208281Z * [new branch] gh/benjaminglass1/108/base -> origin/gh/benjaminglass1/108/base 2025-12-04T08:57:44.1209408Z * [new branch] gh/benjaminglass1/108/head -> origin/gh/benjaminglass1/108/head 2025-12-04T08:57:44.1211053Z * [new branch] gh/benjaminglass1/108/orig -> origin/gh/benjaminglass1/108/orig 2025-12-04T08:57:44.1212582Z * [new branch] gh/benjaminglass1/109/base -> origin/gh/benjaminglass1/109/base 2025-12-04T08:57:44.1213643Z * [new branch] gh/benjaminglass1/109/head -> origin/gh/benjaminglass1/109/head 2025-12-04T08:57:44.1214902Z * [new branch] gh/benjaminglass1/109/orig -> origin/gh/benjaminglass1/109/orig 2025-12-04T08:57:44.1216354Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-12-04T08:57:44.1217428Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-12-04T08:57:44.1218702Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-12-04T08:57:44.1220438Z * [new branch] gh/bobrenjc93/570/base -> origin/gh/bobrenjc93/570/base 2025-12-04T08:57:44.1221627Z * [new branch] gh/bobrenjc93/570/head -> origin/gh/bobrenjc93/570/head 2025-12-04T08:57:44.1222746Z * [new branch] gh/bobrenjc93/570/orig -> origin/gh/bobrenjc93/570/orig 2025-12-04T08:57:44.1224556Z * [new branch] gh/bobrenjc93/604/base -> origin/gh/bobrenjc93/604/base 2025-12-04T08:57:44.1226055Z * [new branch] gh/bobrenjc93/604/head -> origin/gh/bobrenjc93/604/head 2025-12-04T08:57:44.1227707Z * [new branch] gh/bobrenjc93/604/orig -> origin/gh/bobrenjc93/604/orig 2025-12-04T08:57:44.1229335Z * [new branch] gh/bobrenjc93/638/base -> origin/gh/bobrenjc93/638/base 2025-12-04T08:57:44.1230603Z * [new branch] gh/bobrenjc93/638/head -> origin/gh/bobrenjc93/638/head 2025-12-04T08:57:44.1231734Z * [new branch] gh/bobrenjc93/638/orig -> origin/gh/bobrenjc93/638/orig 2025-12-04T08:57:44.1233710Z * [new branch] gh/bobrenjc93/653/base -> origin/gh/bobrenjc93/653/base 2025-12-04T08:57:44.1234891Z * [new branch] gh/bobrenjc93/653/head -> origin/gh/bobrenjc93/653/head 2025-12-04T08:57:44.1236012Z * [new branch] gh/bobrenjc93/653/orig -> origin/gh/bobrenjc93/653/orig 2025-12-04T08:57:44.1237650Z * [new branch] gh/bobrenjc93/654/base -> origin/gh/bobrenjc93/654/base 2025-12-04T08:57:44.1238987Z * [new branch] gh/bobrenjc93/654/head -> origin/gh/bobrenjc93/654/head 2025-12-04T08:57:44.1239998Z * [new branch] gh/bobrenjc93/654/orig -> origin/gh/bobrenjc93/654/orig 2025-12-04T08:57:44.1241553Z * [new branch] gh/bobrenjc93/657/base -> origin/gh/bobrenjc93/657/base 2025-12-04T08:57:44.1242598Z * [new branch] gh/bobrenjc93/657/head -> origin/gh/bobrenjc93/657/head 2025-12-04T08:57:44.1243688Z * [new branch] gh/bobrenjc93/657/orig -> origin/gh/bobrenjc93/657/orig 2025-12-04T08:57:44.1245195Z * [new branch] gh/bobrenjc93/672/base -> origin/gh/bobrenjc93/672/base 2025-12-04T08:57:44.1246334Z * [new branch] gh/bobrenjc93/672/head -> origin/gh/bobrenjc93/672/head 2025-12-04T08:57:44.1247453Z * [new branch] gh/bobrenjc93/672/orig -> origin/gh/bobrenjc93/672/orig 2025-12-04T08:57:44.1248917Z * [new branch] gh/bobrenjc93/679/base -> origin/gh/bobrenjc93/679/base 2025-12-04T08:57:44.1250306Z * [new branch] gh/bobrenjc93/679/head -> origin/gh/bobrenjc93/679/head 2025-12-04T08:57:44.1251393Z * [new branch] gh/bobrenjc93/679/orig -> origin/gh/bobrenjc93/679/orig 2025-12-04T08:57:44.1252984Z * [new branch] gh/bobrenjc93/680/base -> origin/gh/bobrenjc93/680/base 2025-12-04T08:57:44.1254021Z * [new branch] gh/bobrenjc93/680/head -> origin/gh/bobrenjc93/680/head 2025-12-04T08:57:44.1255107Z * [new branch] gh/bobrenjc93/680/orig -> origin/gh/bobrenjc93/680/orig 2025-12-04T08:57:44.1256414Z * [new branch] gh/bobrenjc93/681/base -> origin/gh/bobrenjc93/681/base 2025-12-04T08:57:44.1257492Z * [new branch] gh/bobrenjc93/681/head -> origin/gh/bobrenjc93/681/head 2025-12-04T08:57:44.1258589Z * [new branch] gh/bobrenjc93/681/orig -> origin/gh/bobrenjc93/681/orig 2025-12-04T08:57:44.1259929Z * [new branch] gh/bobrenjc93/682/base -> origin/gh/bobrenjc93/682/base 2025-12-04T08:57:44.1261572Z * [new branch] gh/bobrenjc93/682/head -> origin/gh/bobrenjc93/682/head 2025-12-04T08:57:44.1265630Z * [new branch] gh/bobrenjc93/682/orig -> origin/gh/bobrenjc93/682/orig 2025-12-04T08:57:44.1266289Z * [new branch] gh/bobrenjc93/683/base -> origin/gh/bobrenjc93/683/base 2025-12-04T08:57:44.1266946Z * [new branch] gh/bobrenjc93/683/head -> origin/gh/bobrenjc93/683/head 2025-12-04T08:57:44.1267868Z * [new branch] gh/bobrenjc93/683/orig -> origin/gh/bobrenjc93/683/orig 2025-12-04T08:57:44.1268541Z * [new branch] gh/bobrenjc93/684/base -> origin/gh/bobrenjc93/684/base 2025-12-04T08:57:44.1269195Z * [new branch] gh/bobrenjc93/684/head -> origin/gh/bobrenjc93/684/head 2025-12-04T08:57:44.1270482Z * [new branch] gh/bobrenjc93/684/orig -> origin/gh/bobrenjc93/684/orig 2025-12-04T08:57:44.1271896Z * [new branch] gh/bobrenjc93/685/base -> origin/gh/bobrenjc93/685/base 2025-12-04T08:57:44.1273294Z * [new branch] gh/bobrenjc93/685/head -> origin/gh/bobrenjc93/685/head 2025-12-04T08:57:44.1274709Z * [new branch] gh/bobrenjc93/685/orig -> origin/gh/bobrenjc93/685/orig 2025-12-04T08:57:44.1276462Z * [new branch] gh/bobrenjc93/686/base -> origin/gh/bobrenjc93/686/base 2025-12-04T08:57:44.1279738Z * [new branch] gh/bobrenjc93/686/head -> origin/gh/bobrenjc93/686/head 2025-12-04T08:57:44.1281091Z * [new branch] gh/bobrenjc93/686/orig -> origin/gh/bobrenjc93/686/orig 2025-12-04T08:57:44.1281719Z * [new branch] gh/bobrenjc93/687/base -> origin/gh/bobrenjc93/687/base 2025-12-04T08:57:44.1282364Z * [new branch] gh/bobrenjc93/687/head -> origin/gh/bobrenjc93/687/head 2025-12-04T08:57:44.1283137Z * [new branch] gh/bobrenjc93/687/orig -> origin/gh/bobrenjc93/687/orig 2025-12-04T08:57:44.1284871Z * [new branch] gh/bobrenjc93/688/base -> origin/gh/bobrenjc93/688/base 2025-12-04T08:57:44.1286358Z * [new branch] gh/bobrenjc93/688/head -> origin/gh/bobrenjc93/688/head 2025-12-04T08:57:44.1287405Z * [new branch] gh/bobrenjc93/688/orig -> origin/gh/bobrenjc93/688/orig 2025-12-04T08:57:44.1288832Z * [new branch] gh/bobrenjc93/689/base -> origin/gh/bobrenjc93/689/base 2025-12-04T08:57:44.1290008Z * [new branch] gh/bobrenjc93/689/head -> origin/gh/bobrenjc93/689/head 2025-12-04T08:57:44.1291175Z * [new branch] gh/bobrenjc93/689/orig -> origin/gh/bobrenjc93/689/orig 2025-12-04T08:57:44.1292580Z * [new branch] gh/bobrenjc93/690/base -> origin/gh/bobrenjc93/690/base 2025-12-04T08:57:44.1293792Z * [new branch] gh/bobrenjc93/690/head -> origin/gh/bobrenjc93/690/head 2025-12-04T08:57:44.1294900Z * [new branch] gh/bobrenjc93/690/orig -> origin/gh/bobrenjc93/690/orig 2025-12-04T08:57:44.1297200Z * [new branch] gh/bobrenjc93/691/base -> origin/gh/bobrenjc93/691/base 2025-12-04T08:57:44.1298612Z * [new branch] gh/bobrenjc93/691/head -> origin/gh/bobrenjc93/691/head 2025-12-04T08:57:44.1300208Z * [new branch] gh/bobrenjc93/691/orig -> origin/gh/bobrenjc93/691/orig 2025-12-04T08:57:44.1302448Z * [new branch] gh/bobrenjc93/692/base -> origin/gh/bobrenjc93/692/base 2025-12-04T08:57:44.1303533Z * [new branch] gh/bobrenjc93/692/head -> origin/gh/bobrenjc93/692/head 2025-12-04T08:57:44.1304646Z * [new branch] gh/bobrenjc93/692/orig -> origin/gh/bobrenjc93/692/orig 2025-12-04T08:57:44.1306072Z * [new branch] gh/bobrenjc93/693/base -> origin/gh/bobrenjc93/693/base 2025-12-04T08:57:44.1307133Z * [new branch] gh/bobrenjc93/693/head -> origin/gh/bobrenjc93/693/head 2025-12-04T08:57:44.1308615Z * [new branch] gh/bobrenjc93/693/orig -> origin/gh/bobrenjc93/693/orig 2025-12-04T08:57:44.1310259Z * [new branch] gh/bobrenjc93/694/base -> origin/gh/bobrenjc93/694/base 2025-12-04T08:57:44.1311489Z * [new branch] gh/bobrenjc93/694/head -> origin/gh/bobrenjc93/694/head 2025-12-04T08:57:44.1312611Z * [new branch] gh/bobrenjc93/694/orig -> origin/gh/bobrenjc93/694/orig 2025-12-04T08:57:44.1314092Z * [new branch] gh/bobrenjc93/695/base -> origin/gh/bobrenjc93/695/base 2025-12-04T08:57:44.1315203Z * [new branch] gh/bobrenjc93/695/head -> origin/gh/bobrenjc93/695/head 2025-12-04T08:57:44.1316340Z * [new branch] gh/bobrenjc93/695/orig -> origin/gh/bobrenjc93/695/orig 2025-12-04T08:57:44.1318252Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-12-04T08:57:44.1319342Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-12-04T08:57:44.1321102Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-12-04T08:57:44.1322095Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-12-04T08:57:44.1323136Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-12-04T08:57:44.1325038Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-12-04T08:57:44.1326198Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-12-04T08:57:44.1327478Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-12-04T08:57:44.1328957Z * [new branch] gh/c00w/56/base -> origin/gh/c00w/56/base 2025-12-04T08:57:44.1330036Z * [new branch] gh/c00w/56/head -> origin/gh/c00w/56/head 2025-12-04T08:57:44.1331134Z * [new branch] gh/c00w/56/orig -> origin/gh/c00w/56/orig 2025-12-04T08:57:44.1332825Z * [new branch] gh/c00w/57/base -> origin/gh/c00w/57/base 2025-12-04T08:57:44.1333788Z * [new branch] gh/c00w/57/head -> origin/gh/c00w/57/head 2025-12-04T08:57:44.1334914Z * [new branch] gh/c00w/57/orig -> origin/gh/c00w/57/orig 2025-12-04T08:57:44.1336575Z * [new branch] gh/c00w/58/base -> origin/gh/c00w/58/base 2025-12-04T08:57:44.1337639Z * [new branch] gh/c00w/58/head -> origin/gh/c00w/58/head 2025-12-04T08:57:44.1338711Z * [new branch] gh/c00w/58/orig -> origin/gh/c00w/58/orig 2025-12-04T08:57:44.1340722Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-12-04T08:57:44.1341854Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-12-04T08:57:44.1342962Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-12-04T08:57:44.1344901Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-12-04T08:57:44.1346109Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-12-04T08:57:44.1348238Z * [new branch] gh/coconutruben/55/base -> origin/gh/coconutruben/55/base 2025-12-04T08:57:44.1349282Z * [new branch] gh/coconutruben/55/head -> origin/gh/coconutruben/55/head 2025-12-04T08:57:44.1350485Z * [new branch] gh/coconutruben/55/orig -> origin/gh/coconutruben/55/orig 2025-12-04T08:57:44.1352231Z * [new branch] gh/coconutruben/57/base -> origin/gh/coconutruben/57/base 2025-12-04T08:57:44.1353494Z * [new branch] gh/coconutruben/57/head -> origin/gh/coconutruben/57/head 2025-12-04T08:57:44.1354699Z * [new branch] gh/coconutruben/57/orig -> origin/gh/coconutruben/57/orig 2025-12-04T08:57:44.1356485Z * [new branch] gh/coconutruben/70/base -> origin/gh/coconutruben/70/base 2025-12-04T08:57:44.1357726Z * [new branch] gh/coconutruben/70/head -> origin/gh/coconutruben/70/head 2025-12-04T08:57:44.1358923Z * [new branch] gh/coconutruben/70/orig -> origin/gh/coconutruben/70/orig 2025-12-04T08:57:44.1360478Z * [new branch] gh/coconutruben/71/base -> origin/gh/coconutruben/71/base 2025-12-04T08:57:44.1361570Z * [new branch] gh/coconutruben/71/head -> origin/gh/coconutruben/71/head 2025-12-04T08:57:44.1362736Z * [new branch] gh/coconutruben/71/orig -> origin/gh/coconutruben/71/orig 2025-12-04T08:57:44.1364110Z * [new branch] gh/coconutruben/72/base -> origin/gh/coconutruben/72/base 2025-12-04T08:57:44.1365206Z * [new branch] gh/coconutruben/72/head -> origin/gh/coconutruben/72/head 2025-12-04T08:57:44.1366325Z * [new branch] gh/coconutruben/72/orig -> origin/gh/coconutruben/72/orig 2025-12-04T08:57:44.1367863Z * [new branch] gh/coconutruben/73/base -> origin/gh/coconutruben/73/base 2025-12-04T08:57:44.1368790Z * [new branch] gh/coconutruben/73/head -> origin/gh/coconutruben/73/head 2025-12-04T08:57:44.1369897Z * [new branch] gh/coconutruben/73/orig -> origin/gh/coconutruben/73/orig 2025-12-04T08:57:44.1371731Z * [new branch] gh/coconutruben/74/base -> origin/gh/coconutruben/74/base 2025-12-04T08:57:44.1372882Z * [new branch] gh/coconutruben/74/head -> origin/gh/coconutruben/74/head 2025-12-04T08:57:44.1374011Z * [new branch] gh/coconutruben/74/orig -> origin/gh/coconutruben/74/orig 2025-12-04T08:57:44.1375591Z * [new branch] gh/coconutruben/79/base -> origin/gh/coconutruben/79/base 2025-12-04T08:57:44.1376763Z * [new branch] gh/coconutruben/79/head -> origin/gh/coconutruben/79/head 2025-12-04T08:57:44.1378041Z * [new branch] gh/coconutruben/79/orig -> origin/gh/coconutruben/79/orig 2025-12-04T08:57:44.1379569Z * [new branch] gh/coconutruben/80/base -> origin/gh/coconutruben/80/base 2025-12-04T08:57:44.1380686Z * [new branch] gh/coconutruben/80/head -> origin/gh/coconutruben/80/head 2025-12-04T08:57:44.1381856Z * [new branch] gh/coconutruben/80/orig -> origin/gh/coconutruben/80/orig 2025-12-04T08:57:44.1383559Z * [new branch] gh/coconutruben/82/base -> origin/gh/coconutruben/82/base 2025-12-04T08:57:44.1384579Z * [new branch] gh/coconutruben/82/head -> origin/gh/coconutruben/82/head 2025-12-04T08:57:44.1385664Z * [new branch] gh/coconutruben/82/orig -> origin/gh/coconutruben/82/orig 2025-12-04T08:57:44.1387511Z * [new branch] gh/coconutruben/83/base -> origin/gh/coconutruben/83/base 2025-12-04T08:57:44.1388772Z * [new branch] gh/coconutruben/83/head -> origin/gh/coconutruben/83/head 2025-12-04T08:57:44.1389931Z * [new branch] gh/coconutruben/83/orig -> origin/gh/coconutruben/83/orig 2025-12-04T08:57:44.1391842Z * [new branch] gh/coconutruben/84/base -> origin/gh/coconutruben/84/base 2025-12-04T08:57:44.1392754Z * [new branch] gh/coconutruben/84/head -> origin/gh/coconutruben/84/head 2025-12-04T08:57:44.1393889Z * [new branch] gh/coconutruben/84/orig -> origin/gh/coconutruben/84/orig 2025-12-04T08:57:44.1395480Z * [new branch] gh/coconutruben/85/base -> origin/gh/coconutruben/85/base 2025-12-04T08:57:44.1396661Z * [new branch] gh/coconutruben/85/head -> origin/gh/coconutruben/85/head 2025-12-04T08:57:44.1397775Z * [new branch] gh/coconutruben/85/orig -> origin/gh/coconutruben/85/orig 2025-12-04T08:57:44.1399410Z * [new branch] gh/coconutruben/86/base -> origin/gh/coconutruben/86/base 2025-12-04T08:57:44.1400594Z * [new branch] gh/coconutruben/86/head -> origin/gh/coconutruben/86/head 2025-12-04T08:57:44.1401724Z * [new branch] gh/coconutruben/86/orig -> origin/gh/coconutruben/86/orig 2025-12-04T08:57:44.1403712Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-12-04T08:57:44.1404781Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-12-04T08:57:44.1406203Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-12-04T08:57:44.1407210Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-12-04T08:57:44.1408583Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-12-04T08:57:44.1409536Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-12-04T08:57:44.1410903Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-12-04T08:57:44.1411970Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-12-04T08:57:44.1413768Z * [new branch] gh/d4l3k/1/base -> origin/gh/d4l3k/1/base 2025-12-04T08:57:44.1414813Z * [new branch] gh/d4l3k/1/head -> origin/gh/d4l3k/1/head 2025-12-04T08:57:44.1416382Z * [new branch] gh/d4l3k/2/base -> origin/gh/d4l3k/2/base 2025-12-04T08:57:44.1417426Z * [new branch] gh/d4l3k/2/head -> origin/gh/d4l3k/2/head 2025-12-04T08:57:44.1418598Z * [new branch] gh/d4l3k/2/orig -> origin/gh/d4l3k/2/orig 2025-12-04T08:57:44.1420142Z * [new branch] gh/d4l3k/3/base -> origin/gh/d4l3k/3/base 2025-12-04T08:57:44.1421149Z * [new branch] gh/d4l3k/3/head -> origin/gh/d4l3k/3/head 2025-12-04T08:57:44.1422232Z * [new branch] gh/d4l3k/3/orig -> origin/gh/d4l3k/3/orig 2025-12-04T08:57:44.1424272Z * [new branch] gh/d4l3k/4/base -> origin/gh/d4l3k/4/base 2025-12-04T08:57:44.1425417Z * [new branch] gh/d4l3k/4/head -> origin/gh/d4l3k/4/head 2025-12-04T08:57:44.1426508Z * [new branch] gh/d4l3k/4/orig -> origin/gh/d4l3k/4/orig 2025-12-04T08:57:44.1428199Z * [new branch] gh/d4l3k/5/base -> origin/gh/d4l3k/5/base 2025-12-04T08:57:44.1429299Z * [new branch] gh/d4l3k/5/orig -> origin/gh/d4l3k/5/orig 2025-12-04T08:57:44.1431347Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-12-04T08:57:44.1432414Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-12-04T08:57:44.1433574Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-12-04T08:57:44.1435246Z * [new branch] gh/davidberard98/399/base -> origin/gh/davidberard98/399/base 2025-12-04T08:57:44.1436422Z * [new branch] gh/davidberard98/399/head -> origin/gh/davidberard98/399/head 2025-12-04T08:57:44.1437597Z * [new branch] gh/davidberard98/399/orig -> origin/gh/davidberard98/399/orig 2025-12-04T08:57:44.1439472Z * [new branch] gh/desertfire/605/base -> origin/gh/desertfire/605/base 2025-12-04T08:57:44.1440517Z * [new branch] gh/desertfire/605/head -> origin/gh/desertfire/605/head 2025-12-04T08:57:44.1441675Z * [new branch] gh/desertfire/605/orig -> origin/gh/desertfire/605/orig 2025-12-04T08:57:44.1443167Z * [new branch] gh/desertfire/606/base -> origin/gh/desertfire/606/base 2025-12-04T08:57:44.1444183Z * [new branch] gh/desertfire/606/head -> origin/gh/desertfire/606/head 2025-12-04T08:57:44.1445568Z * [new branch] gh/desertfire/606/orig -> origin/gh/desertfire/606/orig 2025-12-04T08:57:44.1446988Z * [new branch] gh/desertfire/607/base -> origin/gh/desertfire/607/base 2025-12-04T08:57:44.1448045Z * [new branch] gh/desertfire/607/head -> origin/gh/desertfire/607/head 2025-12-04T08:57:44.1449170Z * [new branch] gh/desertfire/607/orig -> origin/gh/desertfire/607/orig 2025-12-04T08:57:44.1450673Z * [new branch] gh/desertfire/608/base -> origin/gh/desertfire/608/base 2025-12-04T08:57:44.1451702Z * [new branch] gh/desertfire/608/head -> origin/gh/desertfire/608/head 2025-12-04T08:57:44.1452876Z * [new branch] gh/desertfire/608/orig -> origin/gh/desertfire/608/orig 2025-12-04T08:57:44.1454375Z * [new branch] gh/desertfire/609/base -> origin/gh/desertfire/609/base 2025-12-04T08:57:44.1455410Z * [new branch] gh/desertfire/609/head -> origin/gh/desertfire/609/head 2025-12-04T08:57:44.1456492Z * [new branch] gh/desertfire/609/orig -> origin/gh/desertfire/609/orig 2025-12-04T08:57:44.1458249Z * [new branch] gh/desertfire/610/base -> origin/gh/desertfire/610/base 2025-12-04T08:57:44.1459381Z * [new branch] gh/desertfire/610/head -> origin/gh/desertfire/610/head 2025-12-04T08:57:44.1460713Z * [new branch] gh/desertfire/610/orig -> origin/gh/desertfire/610/orig 2025-12-04T08:57:44.1462154Z * [new branch] gh/desertfire/611/base -> origin/gh/desertfire/611/base 2025-12-04T08:57:44.1463268Z * [new branch] gh/desertfire/611/head -> origin/gh/desertfire/611/head 2025-12-04T08:57:44.1464416Z * [new branch] gh/desertfire/611/orig -> origin/gh/desertfire/611/orig 2025-12-04T08:57:44.1465952Z * [new branch] gh/desertfire/612/base -> origin/gh/desertfire/612/base 2025-12-04T08:57:44.1467013Z * [new branch] gh/desertfire/612/head -> origin/gh/desertfire/612/head 2025-12-04T08:57:44.1468477Z * [new branch] gh/desertfire/612/orig -> origin/gh/desertfire/612/orig 2025-12-04T08:57:44.1470189Z * [new branch] gh/desertfire/613/base -> origin/gh/desertfire/613/base 2025-12-04T08:57:44.1471241Z * [new branch] gh/desertfire/613/head -> origin/gh/desertfire/613/head 2025-12-04T08:57:44.1472364Z * [new branch] gh/desertfire/613/orig -> origin/gh/desertfire/613/orig 2025-12-04T08:57:44.1474102Z * [new branch] gh/desertfire/614/base -> origin/gh/desertfire/614/base 2025-12-04T08:57:44.1475252Z * [new branch] gh/desertfire/614/head -> origin/gh/desertfire/614/head 2025-12-04T08:57:44.1476540Z * [new branch] gh/desertfire/614/orig -> origin/gh/desertfire/614/orig 2025-12-04T08:57:44.1478214Z * [new branch] gh/desertfire/615/base -> origin/gh/desertfire/615/base 2025-12-04T08:57:44.1479499Z * [new branch] gh/desertfire/615/head -> origin/gh/desertfire/615/head 2025-12-04T08:57:44.1480689Z * [new branch] gh/desertfire/615/orig -> origin/gh/desertfire/615/orig 2025-12-04T08:57:44.1482079Z * [new branch] gh/desertfire/616/base -> origin/gh/desertfire/616/base 2025-12-04T08:57:44.1483200Z * [new branch] gh/desertfire/616/head -> origin/gh/desertfire/616/head 2025-12-04T08:57:44.1484216Z * [new branch] gh/desertfire/616/orig -> origin/gh/desertfire/616/orig 2025-12-04T08:57:44.1485604Z * [new branch] gh/desertfire/617/base -> origin/gh/desertfire/617/base 2025-12-04T08:57:44.1486742Z * [new branch] gh/desertfire/617/head -> origin/gh/desertfire/617/head 2025-12-04T08:57:44.1487761Z * [new branch] gh/desertfire/617/orig -> origin/gh/desertfire/617/orig 2025-12-04T08:57:44.1489662Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-12-04T08:57:44.1490861Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-12-04T08:57:44.1492676Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-12-04T08:57:44.1493693Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-12-04T08:57:44.1494785Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-12-04T08:57:44.1496354Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-12-04T08:57:44.1497408Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-12-04T08:57:44.1498766Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-12-04T08:57:44.1499731Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-12-04T08:57:44.1501120Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-12-04T08:57:44.1502148Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-12-04T08:57:44.1503794Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-12-04T08:57:44.1504843Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-12-04T08:57:44.1506311Z * [new branch] gh/drisspg/194/base -> origin/gh/drisspg/194/base 2025-12-04T08:57:44.1507444Z * [new branch] gh/drisspg/194/head -> origin/gh/drisspg/194/head 2025-12-04T08:57:44.1508826Z * [new branch] gh/drisspg/194/orig -> origin/gh/drisspg/194/orig 2025-12-04T08:57:44.1510346Z * [new branch] gh/drisspg/200/base -> origin/gh/drisspg/200/base 2025-12-04T08:57:44.1511383Z * [new branch] gh/drisspg/200/head -> origin/gh/drisspg/200/head 2025-12-04T08:57:44.1512504Z * [new branch] gh/drisspg/200/orig -> origin/gh/drisspg/200/orig 2025-12-04T08:57:44.1514092Z * [new branch] gh/drisspg/218/base -> origin/gh/drisspg/218/base 2025-12-04T08:57:44.1515258Z * [new branch] gh/drisspg/218/head -> origin/gh/drisspg/218/head 2025-12-04T08:57:44.1516276Z * [new branch] gh/drisspg/218/orig -> origin/gh/drisspg/218/orig 2025-12-04T08:57:44.1517836Z * [new branch] gh/drisspg/219/base -> origin/gh/drisspg/219/base 2025-12-04T08:57:44.1519044Z * [new branch] gh/drisspg/219/head -> origin/gh/drisspg/219/head 2025-12-04T08:57:44.1520288Z * [new branch] gh/drisspg/219/orig -> origin/gh/drisspg/219/orig 2025-12-04T08:57:44.1521756Z * [new branch] gh/drisspg/220/base -> origin/gh/drisspg/220/base 2025-12-04T08:57:44.1522828Z * [new branch] gh/drisspg/220/head -> origin/gh/drisspg/220/head 2025-12-04T08:57:44.1524104Z * [new branch] gh/drisspg/220/orig -> origin/gh/drisspg/220/orig 2025-12-04T08:57:44.1525939Z * [new branch] gh/drisspg/221/base -> origin/gh/drisspg/221/base 2025-12-04T08:57:44.1527006Z * [new branch] gh/drisspg/221/head -> origin/gh/drisspg/221/head 2025-12-04T08:57:44.1528116Z * [new branch] gh/drisspg/221/orig -> origin/gh/drisspg/221/orig 2025-12-04T08:57:44.1529668Z * [new branch] gh/drisspg/222/base -> origin/gh/drisspg/222/base 2025-12-04T08:57:44.1530755Z * [new branch] gh/drisspg/222/head -> origin/gh/drisspg/222/head 2025-12-04T08:57:44.1531911Z * [new branch] gh/drisspg/222/orig -> origin/gh/drisspg/222/orig 2025-12-04T08:57:44.1533410Z * [new branch] gh/drisspg/223/base -> origin/gh/drisspg/223/base 2025-12-04T08:57:44.1534594Z * [new branch] gh/drisspg/223/head -> origin/gh/drisspg/223/head 2025-12-04T08:57:44.1535738Z * [new branch] gh/drisspg/223/orig -> origin/gh/drisspg/223/orig 2025-12-04T08:57:44.1537343Z * [new branch] gh/drisspg/224/base -> origin/gh/drisspg/224/base 2025-12-04T08:57:44.1538347Z * [new branch] gh/drisspg/224/head -> origin/gh/drisspg/224/head 2025-12-04T08:57:44.1539449Z * [new branch] gh/drisspg/224/orig -> origin/gh/drisspg/224/orig 2025-12-04T08:57:44.1540953Z * [new branch] gh/drisspg/225/base -> origin/gh/drisspg/225/base 2025-12-04T08:57:44.1541978Z * [new branch] gh/drisspg/225/head -> origin/gh/drisspg/225/head 2025-12-04T08:57:44.1543110Z * [new branch] gh/drisspg/225/orig -> origin/gh/drisspg/225/orig 2025-12-04T08:57:44.1544600Z * [new branch] gh/drisspg/226/base -> origin/gh/drisspg/226/base 2025-12-04T08:57:44.1545641Z * [new branch] gh/drisspg/226/head -> origin/gh/drisspg/226/head 2025-12-04T08:57:44.1546719Z * [new branch] gh/drisspg/226/orig -> origin/gh/drisspg/226/orig 2025-12-04T08:57:44.1549007Z * [new branch] gh/drisspg/227/base -> origin/gh/drisspg/227/base 2025-12-04T08:57:44.1550200Z * [new branch] gh/drisspg/227/head -> origin/gh/drisspg/227/head 2025-12-04T08:57:44.1551319Z * [new branch] gh/drisspg/227/orig -> origin/gh/drisspg/227/orig 2025-12-04T08:57:44.1552890Z * [new branch] gh/drisspg/228/base -> origin/gh/drisspg/228/base 2025-12-04T08:57:44.1554012Z * [new branch] gh/drisspg/228/head -> origin/gh/drisspg/228/head 2025-12-04T08:57:44.1555239Z * [new branch] gh/drisspg/228/orig -> origin/gh/drisspg/228/orig 2025-12-04T08:57:44.1556769Z * [new branch] gh/drisspg/229/base -> origin/gh/drisspg/229/base 2025-12-04T08:57:44.1557869Z * [new branch] gh/drisspg/229/head -> origin/gh/drisspg/229/head 2025-12-04T08:57:44.1558973Z * [new branch] gh/drisspg/229/orig -> origin/gh/drisspg/229/orig 2025-12-04T08:57:44.1560808Z * [new branch] gh/drisspg/230/base -> origin/gh/drisspg/230/base 2025-12-04T08:57:44.1561743Z * [new branch] gh/drisspg/230/head -> origin/gh/drisspg/230/head 2025-12-04T08:57:44.1562844Z * [new branch] gh/drisspg/230/orig -> origin/gh/drisspg/230/orig 2025-12-04T08:57:44.1564831Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-12-04T08:57:44.1565874Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-12-04T08:57:44.1567701Z * [new branch] gh/dzmitry-huba/1/base -> origin/gh/dzmitry-huba/1/base 2025-12-04T08:57:44.1568795Z * [new branch] gh/dzmitry-huba/1/head -> origin/gh/dzmitry-huba/1/head 2025-12-04T08:57:44.1570432Z * [new branch] gh/dzmitry-huba/12/base -> origin/gh/dzmitry-huba/12/base 2025-12-04T08:57:44.1571658Z * [new branch] gh/dzmitry-huba/12/head -> origin/gh/dzmitry-huba/12/head 2025-12-04T08:57:44.1572814Z * [new branch] gh/dzmitry-huba/12/orig -> origin/gh/dzmitry-huba/12/orig 2025-12-04T08:57:44.1574442Z * [new branch] gh/dzmitry-huba/13/base -> origin/gh/dzmitry-huba/13/base 2025-12-04T08:57:44.1575541Z * [new branch] gh/dzmitry-huba/13/head -> origin/gh/dzmitry-huba/13/head 2025-12-04T08:57:44.1576706Z * [new branch] gh/dzmitry-huba/13/orig -> origin/gh/dzmitry-huba/13/orig 2025-12-04T08:57:44.1578157Z * [new branch] gh/dzmitry-huba/14/base -> origin/gh/dzmitry-huba/14/base 2025-12-04T08:57:44.1579350Z * [new branch] gh/dzmitry-huba/14/head -> origin/gh/dzmitry-huba/14/head 2025-12-04T08:57:44.1580453Z * [new branch] gh/dzmitry-huba/14/orig -> origin/gh/dzmitry-huba/14/orig 2025-12-04T08:57:44.1582153Z * [new branch] gh/dzmitry-huba/15/base -> origin/gh/dzmitry-huba/15/base 2025-12-04T08:57:44.1583256Z * [new branch] gh/dzmitry-huba/15/head -> origin/gh/dzmitry-huba/15/head 2025-12-04T08:57:44.1584296Z * [new branch] gh/dzmitry-huba/15/orig -> origin/gh/dzmitry-huba/15/orig 2025-12-04T08:57:44.1585984Z * [new branch] gh/dzmitry-huba/16/base -> origin/gh/dzmitry-huba/16/base 2025-12-04T08:57:44.1587131Z * [new branch] gh/dzmitry-huba/16/head -> origin/gh/dzmitry-huba/16/head 2025-12-04T08:57:44.1588810Z * [new branch] gh/dzmitry-huba/16/orig -> origin/gh/dzmitry-huba/16/orig 2025-12-04T08:57:44.1590255Z * [new branch] gh/dzmitry-huba/17/base -> origin/gh/dzmitry-huba/17/base 2025-12-04T08:57:44.1591331Z * [new branch] gh/dzmitry-huba/17/head -> origin/gh/dzmitry-huba/17/head 2025-12-04T08:57:44.1593076Z * [new branch] gh/dzmitry-huba/17/orig -> origin/gh/dzmitry-huba/17/orig 2025-12-04T08:57:44.1594467Z * [new branch] gh/dzmitry-huba/2/base -> origin/gh/dzmitry-huba/2/base 2025-12-04T08:57:44.1595678Z * [new branch] gh/dzmitry-huba/2/head -> origin/gh/dzmitry-huba/2/head 2025-12-04T08:57:44.1597055Z * [new branch] gh/dzmitry-huba/3/base -> origin/gh/dzmitry-huba/3/base 2025-12-04T08:57:44.1598033Z * [new branch] gh/dzmitry-huba/3/head -> origin/gh/dzmitry-huba/3/head 2025-12-04T08:57:44.1600149Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-12-04T08:57:44.1601255Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-12-04T08:57:44.1602384Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-12-04T08:57:44.1604232Z * [new branch] gh/eellison/822/base -> origin/gh/eellison/822/base 2025-12-04T08:57:44.1605342Z * [new branch] gh/eellison/822/head -> origin/gh/eellison/822/head 2025-12-04T08:57:44.1606570Z * [new branch] gh/eellison/822/orig -> origin/gh/eellison/822/orig 2025-12-04T08:57:44.1608034Z * [new branch] gh/eellison/823/base -> origin/gh/eellison/823/base 2025-12-04T08:57:44.1609078Z * [new branch] gh/eellison/823/head -> origin/gh/eellison/823/head 2025-12-04T08:57:44.1610262Z * [new branch] gh/eellison/823/orig -> origin/gh/eellison/823/orig 2025-12-04T08:57:44.1611758Z * [new branch] gh/eellison/862/base -> origin/gh/eellison/862/base 2025-12-04T08:57:44.1612789Z * [new branch] gh/eellison/862/head -> origin/gh/eellison/862/head 2025-12-04T08:57:44.1613900Z * [new branch] gh/eellison/862/orig -> origin/gh/eellison/862/orig 2025-12-04T08:57:44.1615392Z * [new branch] gh/eellison/863/base -> origin/gh/eellison/863/base 2025-12-04T08:57:44.1616418Z * [new branch] gh/eellison/863/head -> origin/gh/eellison/863/head 2025-12-04T08:57:44.1617532Z * [new branch] gh/eellison/863/orig -> origin/gh/eellison/863/orig 2025-12-04T08:57:44.1618943Z * [new branch] gh/eellison/864/base -> origin/gh/eellison/864/base 2025-12-04T08:57:44.1620037Z * [new branch] gh/eellison/864/head -> origin/gh/eellison/864/head 2025-12-04T08:57:44.1621212Z * [new branch] gh/eellison/864/orig -> origin/gh/eellison/864/orig 2025-12-04T08:57:44.1622887Z * [new branch] gh/eellison/865/base -> origin/gh/eellison/865/base 2025-12-04T08:57:44.1624280Z * [new branch] gh/eellison/865/head -> origin/gh/eellison/865/head 2025-12-04T08:57:44.1627101Z * [new branch] gh/eellison/865/orig -> origin/gh/eellison/865/orig 2025-12-04T08:57:44.1628789Z * [new branch] gh/eellison/866/base -> origin/gh/eellison/866/base 2025-12-04T08:57:44.1629874Z * [new branch] gh/eellison/866/head -> origin/gh/eellison/866/head 2025-12-04T08:57:44.1631000Z * [new branch] gh/eellison/866/orig -> origin/gh/eellison/866/orig 2025-12-04T08:57:44.1632864Z * [new branch] gh/eellison/867/base -> origin/gh/eellison/867/base 2025-12-04T08:57:44.1633773Z * [new branch] gh/eellison/867/head -> origin/gh/eellison/867/head 2025-12-04T08:57:44.1634921Z * [new branch] gh/eellison/867/orig -> origin/gh/eellison/867/orig 2025-12-04T08:57:44.1636697Z * [new branch] gh/eellison/868/base -> origin/gh/eellison/868/base 2025-12-04T08:57:44.1638162Z * [new branch] gh/eellison/868/head -> origin/gh/eellison/868/head 2025-12-04T08:57:44.1639327Z * [new branch] gh/eellison/868/orig -> origin/gh/eellison/868/orig 2025-12-04T08:57:44.1640868Z * [new branch] gh/eellison/869/base -> origin/gh/eellison/869/base 2025-12-04T08:57:44.1641894Z * [new branch] gh/eellison/869/head -> origin/gh/eellison/869/head 2025-12-04T08:57:44.1643060Z * [new branch] gh/eellison/869/orig -> origin/gh/eellison/869/orig 2025-12-04T08:57:44.1644549Z * [new branch] gh/eellison/870/base -> origin/gh/eellison/870/base 2025-12-04T08:57:44.1645535Z * [new branch] gh/eellison/870/head -> origin/gh/eellison/870/head 2025-12-04T08:57:44.1646616Z * [new branch] gh/eellison/870/orig -> origin/gh/eellison/870/orig 2025-12-04T08:57:44.1648212Z * [new branch] gh/eellison/871/base -> origin/gh/eellison/871/base 2025-12-04T08:57:44.1649197Z * [new branch] gh/eellison/871/head -> origin/gh/eellison/871/head 2025-12-04T08:57:44.1650896Z * [new branch] gh/eellison/871/orig -> origin/gh/eellison/871/orig 2025-12-04T08:57:44.1652395Z * [new branch] gh/eellison/872/base -> origin/gh/eellison/872/base 2025-12-04T08:57:44.1653597Z * [new branch] gh/eellison/872/head -> origin/gh/eellison/872/head 2025-12-04T08:57:44.1654552Z * [new branch] gh/eellison/872/orig -> origin/gh/eellison/872/orig 2025-12-04T08:57:44.1656273Z * [new branch] gh/eellison/873/base -> origin/gh/eellison/873/base 2025-12-04T08:57:44.1657278Z * [new branch] gh/eellison/873/head -> origin/gh/eellison/873/head 2025-12-04T08:57:44.1658480Z * [new branch] gh/eellison/873/orig -> origin/gh/eellison/873/orig 2025-12-04T08:57:44.1659994Z * [new branch] gh/eellison/874/base -> origin/gh/eellison/874/base 2025-12-04T08:57:44.1661000Z * [new branch] gh/eellison/874/head -> origin/gh/eellison/874/head 2025-12-04T08:57:44.1662131Z * [new branch] gh/eellison/874/orig -> origin/gh/eellison/874/orig 2025-12-04T08:57:44.1664117Z * [new branch] gh/eellison/875/base -> origin/gh/eellison/875/base 2025-12-04T08:57:44.1665429Z * [new branch] gh/eellison/875/head -> origin/gh/eellison/875/head 2025-12-04T08:57:44.1666456Z * [new branch] gh/eellison/875/orig -> origin/gh/eellison/875/orig 2025-12-04T08:57:44.1668413Z * [new branch] gh/eellison/876/base -> origin/gh/eellison/876/base 2025-12-04T08:57:44.1669483Z * [new branch] gh/eellison/876/head -> origin/gh/eellison/876/head 2025-12-04T08:57:44.1670674Z * [new branch] gh/eellison/876/orig -> origin/gh/eellison/876/orig 2025-12-04T08:57:44.1672301Z * [new branch] gh/eellison/877/base -> origin/gh/eellison/877/base 2025-12-04T08:57:44.1673348Z * [new branch] gh/eellison/877/head -> origin/gh/eellison/877/head 2025-12-04T08:57:44.1674998Z * [new branch] gh/eellison/877/orig -> origin/gh/eellison/877/orig 2025-12-04T08:57:44.1676195Z * [new branch] gh/eellison/878/base -> origin/gh/eellison/878/base 2025-12-04T08:57:44.1677266Z * [new branch] gh/eellison/878/head -> origin/gh/eellison/878/head 2025-12-04T08:57:44.1678431Z * [new branch] gh/eellison/878/orig -> origin/gh/eellison/878/orig 2025-12-04T08:57:44.1680166Z * [new branch] gh/eellison/879/base -> origin/gh/eellison/879/base 2025-12-04T08:57:44.1681194Z * [new branch] gh/eellison/879/head -> origin/gh/eellison/879/head 2025-12-04T08:57:44.1682316Z * [new branch] gh/eellison/879/orig -> origin/gh/eellison/879/orig 2025-12-04T08:57:44.1683720Z * [new branch] gh/eellison/880/base -> origin/gh/eellison/880/base 2025-12-04T08:57:44.1684836Z * [new branch] gh/eellison/880/head -> origin/gh/eellison/880/head 2025-12-04T08:57:44.1685980Z * [new branch] gh/eellison/880/orig -> origin/gh/eellison/880/orig 2025-12-04T08:57:44.1687561Z * [new branch] gh/eellison/881/base -> origin/gh/eellison/881/base 2025-12-04T08:57:44.1688584Z * [new branch] gh/eellison/881/head -> origin/gh/eellison/881/head 2025-12-04T08:57:44.1689795Z * [new branch] gh/eellison/881/orig -> origin/gh/eellison/881/orig 2025-12-04T08:57:44.1691473Z * [new branch] gh/eellison/882/base -> origin/gh/eellison/882/base 2025-12-04T08:57:44.1692548Z * [new branch] gh/eellison/882/head -> origin/gh/eellison/882/head 2025-12-04T08:57:44.1693855Z * [new branch] gh/eellison/882/orig -> origin/gh/eellison/882/orig 2025-12-04T08:57:44.1695294Z * [new branch] gh/eellison/883/base -> origin/gh/eellison/883/base 2025-12-04T08:57:44.1696361Z * [new branch] gh/eellison/883/head -> origin/gh/eellison/883/head 2025-12-04T08:57:44.1697390Z * [new branch] gh/eellison/883/orig -> origin/gh/eellison/883/orig 2025-12-04T08:57:44.1698878Z * [new branch] gh/eellison/884/base -> origin/gh/eellison/884/base 2025-12-04T08:57:44.1699855Z * [new branch] gh/eellison/884/head -> origin/gh/eellison/884/head 2025-12-04T08:57:44.1700917Z * [new branch] gh/eellison/884/orig -> origin/gh/eellison/884/orig 2025-12-04T08:57:44.1702732Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-12-04T08:57:44.1703881Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-12-04T08:57:44.1705649Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-12-04T08:57:44.1706767Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-12-04T08:57:44.1708147Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-12-04T08:57:44.1710103Z * [new branch] gh/etaf/156/base -> origin/gh/etaf/156/base 2025-12-04T08:57:44.1711216Z * [new branch] gh/etaf/156/head -> origin/gh/etaf/156/head 2025-12-04T08:57:44.1712345Z * [new branch] gh/etaf/156/orig -> origin/gh/etaf/156/orig 2025-12-04T08:57:44.1714172Z * [new branch] gh/etaf/157/base -> origin/gh/etaf/157/base 2025-12-04T08:57:44.1715273Z * [new branch] gh/etaf/157/head -> origin/gh/etaf/157/head 2025-12-04T08:57:44.1716414Z * [new branch] gh/etaf/157/orig -> origin/gh/etaf/157/orig 2025-12-04T08:57:44.1717926Z * [new branch] gh/etaf/158/base -> origin/gh/etaf/158/base 2025-12-04T08:57:44.1719085Z * [new branch] gh/etaf/158/head -> origin/gh/etaf/158/head 2025-12-04T08:57:44.1720525Z * [new branch] gh/etaf/158/orig -> origin/gh/etaf/158/orig 2025-12-04T08:57:44.1722075Z * [new branch] gh/etaf/159/base -> origin/gh/etaf/159/base 2025-12-04T08:57:44.1723208Z * [new branch] gh/etaf/159/head -> origin/gh/etaf/159/head 2025-12-04T08:57:44.1724895Z * [new branch] gh/etaf/159/orig -> origin/gh/etaf/159/orig 2025-12-04T08:57:44.1726484Z * [new branch] gh/etaf/160/base -> origin/gh/etaf/160/base 2025-12-04T08:57:44.1727587Z * [new branch] gh/etaf/160/head -> origin/gh/etaf/160/head 2025-12-04T08:57:44.1728753Z * [new branch] gh/etaf/160/orig -> origin/gh/etaf/160/orig 2025-12-04T08:57:44.1730305Z * [new branch] gh/etaf/161/base -> origin/gh/etaf/161/base 2025-12-04T08:57:44.1731631Z * [new branch] gh/etaf/161/head -> origin/gh/etaf/161/head 2025-12-04T08:57:44.1732662Z * [new branch] gh/etaf/161/orig -> origin/gh/etaf/161/orig 2025-12-04T08:57:44.1734239Z * [new branch] gh/etaf/166/base -> origin/gh/etaf/166/base 2025-12-04T08:57:44.1735570Z * [new branch] gh/etaf/166/head -> origin/gh/etaf/166/head 2025-12-04T08:57:44.1736845Z * [new branch] gh/etaf/166/orig -> origin/gh/etaf/166/orig 2025-12-04T08:57:44.1738329Z * [new branch] gh/etaf/167/base -> origin/gh/etaf/167/base 2025-12-04T08:57:44.1739339Z * [new branch] gh/etaf/167/head -> origin/gh/etaf/167/head 2025-12-04T08:57:44.1740475Z * [new branch] gh/etaf/167/orig -> origin/gh/etaf/167/orig 2025-12-04T08:57:44.1742100Z * [new branch] gh/etaf/168/base -> origin/gh/etaf/168/base 2025-12-04T08:57:44.1743244Z * [new branch] gh/etaf/168/head -> origin/gh/etaf/168/head 2025-12-04T08:57:44.1744360Z * [new branch] gh/etaf/168/orig -> origin/gh/etaf/168/orig 2025-12-04T08:57:44.1745958Z * [new branch] gh/etaf/172/base -> origin/gh/etaf/172/base 2025-12-04T08:57:44.1747106Z * [new branch] gh/etaf/172/head -> origin/gh/etaf/172/head 2025-12-04T08:57:44.1748699Z * [new branch] gh/etaf/172/orig -> origin/gh/etaf/172/orig 2025-12-04T08:57:44.1750310Z * [new branch] gh/etaf/173/base -> origin/gh/etaf/173/base 2025-12-04T08:57:44.1751645Z * [new branch] gh/etaf/173/head -> origin/gh/etaf/173/head 2025-12-04T08:57:44.1752814Z * [new branch] gh/etaf/173/orig -> origin/gh/etaf/173/orig 2025-12-04T08:57:44.1754442Z * [new branch] gh/etaf/174/base -> origin/gh/etaf/174/base 2025-12-04T08:57:44.1755540Z * [new branch] gh/etaf/174/head -> origin/gh/etaf/174/head 2025-12-04T08:57:44.1757168Z * [new branch] gh/etaf/175/base -> origin/gh/etaf/175/base 2025-12-04T08:57:44.1758190Z * [new branch] gh/etaf/175/head -> origin/gh/etaf/175/head 2025-12-04T08:57:44.1759416Z * [new branch] gh/etaf/175/orig -> origin/gh/etaf/175/orig 2025-12-04T08:57:44.1761143Z * [new branch] gh/etaf/176/base -> origin/gh/etaf/176/base 2025-12-04T08:57:44.1762283Z * [new branch] gh/etaf/176/head -> origin/gh/etaf/176/head 2025-12-04T08:57:44.1763407Z * [new branch] gh/etaf/176/orig -> origin/gh/etaf/176/orig 2025-12-04T08:57:44.1765380Z * [new branch] gh/etaf/177/base -> origin/gh/etaf/177/base 2025-12-04T08:57:44.1766658Z * [new branch] gh/etaf/177/head -> origin/gh/etaf/177/head 2025-12-04T08:57:44.1767713Z * [new branch] gh/etaf/177/orig -> origin/gh/etaf/177/orig 2025-12-04T08:57:44.1769513Z * [new branch] gh/etaf/178/base -> origin/gh/etaf/178/base 2025-12-04T08:57:44.1770787Z * [new branch] gh/etaf/178/head -> origin/gh/etaf/178/head 2025-12-04T08:57:44.1772092Z * [new branch] gh/etaf/178/orig -> origin/gh/etaf/178/orig 2025-12-04T08:57:44.1773498Z * [new branch] gh/etaf/179/base -> origin/gh/etaf/179/base 2025-12-04T08:57:44.1774516Z * [new branch] gh/etaf/179/head -> origin/gh/etaf/179/head 2025-12-04T08:57:44.1775613Z * [new branch] gh/etaf/179/orig -> origin/gh/etaf/179/orig 2025-12-04T08:57:44.1777030Z * [new branch] gh/etaf/180/base -> origin/gh/etaf/180/base 2025-12-04T08:57:44.1778069Z * [new branch] gh/etaf/180/head -> origin/gh/etaf/180/head 2025-12-04T08:57:44.1779154Z * [new branch] gh/etaf/180/orig -> origin/gh/etaf/180/orig 2025-12-04T08:57:44.1781004Z * [new branch] gh/exclamaforte/1/base -> origin/gh/exclamaforte/1/base 2025-12-04T08:57:44.1782066Z * [new branch] gh/exclamaforte/1/head -> origin/gh/exclamaforte/1/head 2025-12-04T08:57:44.1783630Z * [new branch] gh/exclamaforte/2/base -> origin/gh/exclamaforte/2/base 2025-12-04T08:57:44.1784598Z * [new branch] gh/exclamaforte/2/head -> origin/gh/exclamaforte/2/head 2025-12-04T08:57:44.1786119Z * [new branch] gh/exclamaforte/3/base -> origin/gh/exclamaforte/3/base 2025-12-04T08:57:44.1787313Z * [new branch] gh/exclamaforte/3/head -> origin/gh/exclamaforte/3/head 2025-12-04T08:57:44.1789106Z * [new branch] gh/exclamaforte/4/base -> origin/gh/exclamaforte/4/base 2025-12-04T08:57:44.1790186Z * [new branch] gh/exclamaforte/4/head -> origin/gh/exclamaforte/4/head 2025-12-04T08:57:44.1792157Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-12-04T08:57:44.1793291Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-12-04T08:57:44.1794416Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-12-04T08:57:44.1796100Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-12-04T08:57:44.1797163Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-12-04T08:57:44.1798334Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-12-04T08:57:44.1799974Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-12-04T08:57:44.1801029Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-12-04T08:57:44.1802157Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-12-04T08:57:44.1803615Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-12-04T08:57:44.1804646Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-12-04T08:57:44.1805774Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-12-04T08:57:44.1807269Z * [new branch] gh/ezyang/3139/base -> origin/gh/ezyang/3139/base 2025-12-04T08:57:44.1808286Z * [new branch] gh/ezyang/3139/head -> origin/gh/ezyang/3139/head 2025-12-04T08:57:44.1809410Z * [new branch] gh/ezyang/3139/orig -> origin/gh/ezyang/3139/orig 2025-12-04T08:57:44.1810920Z * [new branch] gh/ezyang/3140/base -> origin/gh/ezyang/3140/base 2025-12-04T08:57:44.1812015Z * [new branch] gh/ezyang/3140/head -> origin/gh/ezyang/3140/head 2025-12-04T08:57:44.1813124Z * [new branch] gh/ezyang/3140/orig -> origin/gh/ezyang/3140/orig 2025-12-04T08:57:44.1814597Z * [new branch] gh/ezyang/3143/base -> origin/gh/ezyang/3143/base 2025-12-04T08:57:44.1815635Z * [new branch] gh/ezyang/3143/head -> origin/gh/ezyang/3143/head 2025-12-04T08:57:44.1816732Z * [new branch] gh/ezyang/3143/orig -> origin/gh/ezyang/3143/orig 2025-12-04T08:57:44.1818230Z * [new branch] gh/ezyang/3144/base -> origin/gh/ezyang/3144/base 2025-12-04T08:57:44.1819318Z * [new branch] gh/ezyang/3144/head -> origin/gh/ezyang/3144/head 2025-12-04T08:57:44.1820436Z * [new branch] gh/ezyang/3144/orig -> origin/gh/ezyang/3144/orig 2025-12-04T08:57:44.1821927Z * [new branch] gh/ezyang/3167/base -> origin/gh/ezyang/3167/base 2025-12-04T08:57:44.1822961Z * [new branch] gh/ezyang/3167/head -> origin/gh/ezyang/3167/head 2025-12-04T08:57:44.1824941Z * [new branch] gh/ezyang/3167/orig -> origin/gh/ezyang/3167/orig 2025-12-04T08:57:44.1826276Z * [new branch] gh/ezyang/3173/base -> origin/gh/ezyang/3173/base 2025-12-04T08:57:44.1827551Z * [new branch] gh/ezyang/3173/head -> origin/gh/ezyang/3173/head 2025-12-04T08:57:44.1828988Z * [new branch] gh/ezyang/3173/orig -> origin/gh/ezyang/3173/orig 2025-12-04T08:57:44.1830460Z * [new branch] gh/ezyang/3175/base -> origin/gh/ezyang/3175/base 2025-12-04T08:57:44.1831498Z * [new branch] gh/ezyang/3175/head -> origin/gh/ezyang/3175/head 2025-12-04T08:57:44.1832630Z * [new branch] gh/ezyang/3175/orig -> origin/gh/ezyang/3175/orig 2025-12-04T08:57:44.1834188Z * [new branch] gh/ezyang/3182/base -> origin/gh/ezyang/3182/base 2025-12-04T08:57:44.1835253Z * [new branch] gh/ezyang/3182/head -> origin/gh/ezyang/3182/head 2025-12-04T08:57:44.1836382Z * [new branch] gh/ezyang/3182/orig -> origin/gh/ezyang/3182/orig 2025-12-04T08:57:44.1837930Z * [new branch] gh/ezyang/3185/base -> origin/gh/ezyang/3185/base 2025-12-04T08:57:44.1839109Z * [new branch] gh/ezyang/3185/head -> origin/gh/ezyang/3185/head 2025-12-04T08:57:44.1840206Z * [new branch] gh/ezyang/3185/orig -> origin/gh/ezyang/3185/orig 2025-12-04T08:57:44.1841872Z * [new branch] gh/ezyang/3189/base -> origin/gh/ezyang/3189/base 2025-12-04T08:57:44.1842824Z * [new branch] gh/ezyang/3189/head -> origin/gh/ezyang/3189/head 2025-12-04T08:57:44.1843935Z * [new branch] gh/ezyang/3189/orig -> origin/gh/ezyang/3189/orig 2025-12-04T08:57:44.1845413Z * [new branch] gh/ezyang/3191/base -> origin/gh/ezyang/3191/base 2025-12-04T08:57:44.1846426Z * [new branch] gh/ezyang/3191/head -> origin/gh/ezyang/3191/head 2025-12-04T08:57:44.1847534Z * [new branch] gh/ezyang/3191/orig -> origin/gh/ezyang/3191/orig 2025-12-04T08:57:44.1849443Z * [new branch] gh/ezyang/3192/base -> origin/gh/ezyang/3192/base 2025-12-04T08:57:44.1850510Z * [new branch] gh/ezyang/3192/head -> origin/gh/ezyang/3192/head 2025-12-04T08:57:44.1851673Z * [new branch] gh/ezyang/3192/orig -> origin/gh/ezyang/3192/orig 2025-12-04T08:57:44.1853265Z * [new branch] gh/ezyang/3193/base -> origin/gh/ezyang/3193/base 2025-12-04T08:57:44.1854294Z * [new branch] gh/ezyang/3193/head -> origin/gh/ezyang/3193/head 2025-12-04T08:57:44.1855457Z * [new branch] gh/ezyang/3193/orig -> origin/gh/ezyang/3193/orig 2025-12-04T08:57:44.1857050Z * [new branch] gh/ezyang/3194/base -> origin/gh/ezyang/3194/base 2025-12-04T08:57:44.1858200Z * [new branch] gh/ezyang/3194/head -> origin/gh/ezyang/3194/head 2025-12-04T08:57:44.1859329Z * [new branch] gh/ezyang/3194/orig -> origin/gh/ezyang/3194/orig 2025-12-04T08:57:44.1860808Z * [new branch] gh/ezyang/3195/base -> origin/gh/ezyang/3195/base 2025-12-04T08:57:44.1861818Z * [new branch] gh/ezyang/3195/head -> origin/gh/ezyang/3195/head 2025-12-04T08:57:44.1862968Z * [new branch] gh/ezyang/3195/orig -> origin/gh/ezyang/3195/orig 2025-12-04T08:57:44.1864462Z * [new branch] gh/ezyang/3196/base -> origin/gh/ezyang/3196/base 2025-12-04T08:57:44.1865495Z * [new branch] gh/ezyang/3196/head -> origin/gh/ezyang/3196/head 2025-12-04T08:57:44.1866616Z * [new branch] gh/ezyang/3196/orig -> origin/gh/ezyang/3196/orig 2025-12-04T08:57:44.1868669Z * [new branch] gh/ezyang/3197/base -> origin/gh/ezyang/3197/base 2025-12-04T08:57:44.1869699Z * [new branch] gh/ezyang/3197/head -> origin/gh/ezyang/3197/head 2025-12-04T08:57:44.1870839Z * [new branch] gh/ezyang/3197/orig -> origin/gh/ezyang/3197/orig 2025-12-04T08:57:44.1872415Z * [new branch] gh/ezyang/3198/base -> origin/gh/ezyang/3198/base 2025-12-04T08:57:44.1873575Z * [new branch] gh/ezyang/3198/head -> origin/gh/ezyang/3198/head 2025-12-04T08:57:44.1874751Z * [new branch] gh/ezyang/3198/orig -> origin/gh/ezyang/3198/orig 2025-12-04T08:57:44.1876311Z * [new branch] gh/ezyang/3199/base -> origin/gh/ezyang/3199/base 2025-12-04T08:57:44.1877399Z * [new branch] gh/ezyang/3199/head -> origin/gh/ezyang/3199/head 2025-12-04T08:57:44.1878582Z * [new branch] gh/ezyang/3199/orig -> origin/gh/ezyang/3199/orig 2025-12-04T08:57:44.1880248Z * [new branch] gh/ezyang/3200/base -> origin/gh/ezyang/3200/base 2025-12-04T08:57:44.1881319Z * [new branch] gh/ezyang/3200/head -> origin/gh/ezyang/3200/head 2025-12-04T08:57:44.1882433Z * [new branch] gh/ezyang/3200/orig -> origin/gh/ezyang/3200/orig 2025-12-04T08:57:44.1883986Z * [new branch] gh/ezyang/3201/base -> origin/gh/ezyang/3201/base 2025-12-04T08:57:44.1885015Z * [new branch] gh/ezyang/3201/head -> origin/gh/ezyang/3201/head 2025-12-04T08:57:44.1886192Z * [new branch] gh/ezyang/3201/orig -> origin/gh/ezyang/3201/orig 2025-12-04T08:57:44.1887639Z * [new branch] gh/ezyang/3202/base -> origin/gh/ezyang/3202/base 2025-12-04T08:57:44.1888759Z * [new branch] gh/ezyang/3202/head -> origin/gh/ezyang/3202/head 2025-12-04T08:57:44.1890098Z * [new branch] gh/ezyang/3202/orig -> origin/gh/ezyang/3202/orig 2025-12-04T08:57:44.1891779Z * [new branch] gh/ezyang/3203/base -> origin/gh/ezyang/3203/base 2025-12-04T08:57:44.1892626Z * [new branch] gh/ezyang/3203/head -> origin/gh/ezyang/3203/head 2025-12-04T08:57:44.1893923Z * [new branch] gh/ezyang/3203/orig -> origin/gh/ezyang/3203/orig 2025-12-04T08:57:44.1895417Z * [new branch] gh/ezyang/3204/base -> origin/gh/ezyang/3204/base 2025-12-04T08:57:44.1896486Z * [new branch] gh/ezyang/3204/head -> origin/gh/ezyang/3204/head 2025-12-04T08:57:44.1897602Z * [new branch] gh/ezyang/3204/orig -> origin/gh/ezyang/3204/orig 2025-12-04T08:57:44.1899159Z * [new branch] gh/ezyang/3205/base -> origin/gh/ezyang/3205/base 2025-12-04T08:57:44.1900191Z * [new branch] gh/ezyang/3205/head -> origin/gh/ezyang/3205/head 2025-12-04T08:57:44.1901258Z * [new branch] gh/ezyang/3205/orig -> origin/gh/ezyang/3205/orig 2025-12-04T08:57:44.1902768Z * [new branch] gh/ezyang/3206/base -> origin/gh/ezyang/3206/base 2025-12-04T08:57:44.1903957Z * [new branch] gh/ezyang/3206/head -> origin/gh/ezyang/3206/head 2025-12-04T08:57:44.1905066Z * [new branch] gh/ezyang/3206/orig -> origin/gh/ezyang/3206/orig 2025-12-04T08:57:44.1906590Z * [new branch] gh/ezyang/3207/base -> origin/gh/ezyang/3207/base 2025-12-04T08:57:44.1907875Z * [new branch] gh/ezyang/3207/head -> origin/gh/ezyang/3207/head 2025-12-04T08:57:44.1909135Z * [new branch] gh/ezyang/3207/orig -> origin/gh/ezyang/3207/orig 2025-12-04T08:57:44.1910732Z * [new branch] gh/ezyang/3208/base -> origin/gh/ezyang/3208/base 2025-12-04T08:57:44.1912339Z * [new branch] gh/ezyang/3208/head -> origin/gh/ezyang/3208/head 2025-12-04T08:57:44.1913453Z * [new branch] gh/ezyang/3208/orig -> origin/gh/ezyang/3208/orig 2025-12-04T08:57:44.1915061Z * [new branch] gh/ezyang/3209/base -> origin/gh/ezyang/3209/base 2025-12-04T08:57:44.1916120Z * [new branch] gh/ezyang/3209/head -> origin/gh/ezyang/3209/head 2025-12-04T08:57:44.1917305Z * [new branch] gh/ezyang/3209/orig -> origin/gh/ezyang/3209/orig 2025-12-04T08:57:44.1919213Z * [new branch] gh/fadara01/3/base -> origin/gh/fadara01/3/base 2025-12-04T08:57:44.1920357Z * [new branch] gh/fadara01/3/head -> origin/gh/fadara01/3/head 2025-12-04T08:57:44.1921504Z * [new branch] gh/fadara01/3/orig -> origin/gh/fadara01/3/orig 2025-12-04T08:57:44.1922990Z * [new branch] gh/fadara01/5/base -> origin/gh/fadara01/5/base 2025-12-04T08:57:44.1924653Z * [new branch] gh/fadara01/5/head -> origin/gh/fadara01/5/head 2025-12-04T08:57:44.1925708Z * [new branch] gh/fadara01/5/orig -> origin/gh/fadara01/5/orig 2025-12-04T08:57:44.1927217Z * [new branch] gh/fadara01/6/base -> origin/gh/fadara01/6/base 2025-12-04T08:57:44.1928347Z * [new branch] gh/fadara01/6/head -> origin/gh/fadara01/6/head 2025-12-04T08:57:44.1929477Z * [new branch] gh/fadara01/6/orig -> origin/gh/fadara01/6/orig 2025-12-04T08:57:44.1931067Z * [new branch] gh/fadara01/7/base -> origin/gh/fadara01/7/base 2025-12-04T08:57:44.1932308Z * [new branch] gh/fadara01/7/head -> origin/gh/fadara01/7/head 2025-12-04T08:57:44.1933354Z * [new branch] gh/fadara01/7/orig -> origin/gh/fadara01/7/orig 2025-12-04T08:57:44.1934962Z * [new branch] gh/fadara01/8/base -> origin/gh/fadara01/8/base 2025-12-04T08:57:44.1936082Z * [new branch] gh/fadara01/8/head -> origin/gh/fadara01/8/head 2025-12-04T08:57:44.1937259Z * [new branch] gh/fadara01/8/orig -> origin/gh/fadara01/8/orig 2025-12-04T08:57:44.1938704Z * [new branch] gh/fadara01/9/base -> origin/gh/fadara01/9/base 2025-12-04T08:57:44.1939816Z * [new branch] gh/fadara01/9/head -> origin/gh/fadara01/9/head 2025-12-04T08:57:44.1940939Z * [new branch] gh/fadara01/9/orig -> origin/gh/fadara01/9/orig 2025-12-04T08:57:44.1942716Z * [new branch] gh/fduwjj/182/base -> origin/gh/fduwjj/182/base 2025-12-04T08:57:44.1943855Z * [new branch] gh/fduwjj/182/head -> origin/gh/fduwjj/182/head 2025-12-04T08:57:44.1944947Z * [new branch] gh/fduwjj/182/orig -> origin/gh/fduwjj/182/orig 2025-12-04T08:57:44.1946412Z * [new branch] gh/fduwjj/211/base -> origin/gh/fduwjj/211/base 2025-12-04T08:57:44.1947832Z * [new branch] gh/fduwjj/211/head -> origin/gh/fduwjj/211/head 2025-12-04T08:57:44.1949072Z * [new branch] gh/fduwjj/211/orig -> origin/gh/fduwjj/211/orig 2025-12-04T08:57:44.1950755Z * [new branch] gh/fduwjj/212/base -> origin/gh/fduwjj/212/base 2025-12-04T08:57:44.1951886Z * [new branch] gh/fduwjj/212/head -> origin/gh/fduwjj/212/head 2025-12-04T08:57:44.1953019Z * [new branch] gh/fduwjj/212/orig -> origin/gh/fduwjj/212/orig 2025-12-04T08:57:44.1954535Z * [new branch] gh/fduwjj/213/base -> origin/gh/fduwjj/213/base 2025-12-04T08:57:44.1955671Z * [new branch] gh/fduwjj/213/head -> origin/gh/fduwjj/213/head 2025-12-04T08:57:44.1957036Z * [new branch] gh/fduwjj/213/orig -> origin/gh/fduwjj/213/orig 2025-12-04T08:57:44.1958712Z * [new branch] gh/fduwjj/226/base -> origin/gh/fduwjj/226/base 2025-12-04T08:57:44.1959883Z * [new branch] gh/fduwjj/226/head -> origin/gh/fduwjj/226/head 2025-12-04T08:57:44.1960961Z * [new branch] gh/fduwjj/226/orig -> origin/gh/fduwjj/226/orig 2025-12-04T08:57:44.1962645Z * [new branch] gh/fduwjj/229/base -> origin/gh/fduwjj/229/base 2025-12-04T08:57:44.1964182Z * [new branch] gh/fduwjj/229/head -> origin/gh/fduwjj/229/head 2025-12-04T08:57:44.1965233Z * [new branch] gh/fduwjj/229/orig -> origin/gh/fduwjj/229/orig 2025-12-04T08:57:44.1966823Z * [new branch] gh/fduwjj/233/base -> origin/gh/fduwjj/233/base 2025-12-04T08:57:44.1967974Z * [new branch] gh/fduwjj/233/head -> origin/gh/fduwjj/233/head 2025-12-04T08:57:44.1969074Z * [new branch] gh/fduwjj/233/orig -> origin/gh/fduwjj/233/orig 2025-12-04T08:57:44.1970558Z * [new branch] gh/fduwjj/234/base -> origin/gh/fduwjj/234/base 2025-12-04T08:57:44.1971748Z * [new branch] gh/fduwjj/234/head -> origin/gh/fduwjj/234/head 2025-12-04T08:57:44.1972780Z * [new branch] gh/fduwjj/234/orig -> origin/gh/fduwjj/234/orig 2025-12-04T08:57:44.1974243Z * [new branch] gh/fduwjj/235/base -> origin/gh/fduwjj/235/base 2025-12-04T08:57:44.1975379Z * [new branch] gh/fduwjj/235/head -> origin/gh/fduwjj/235/head 2025-12-04T08:57:44.1976501Z * [new branch] gh/fduwjj/235/orig -> origin/gh/fduwjj/235/orig 2025-12-04T08:57:44.1977827Z * [new branch] gh/fduwjj/236/base -> origin/gh/fduwjj/236/base 2025-12-04T08:57:44.1978991Z * [new branch] gh/fduwjj/236/head -> origin/gh/fduwjj/236/head 2025-12-04T08:57:44.1980027Z * [new branch] gh/fduwjj/236/orig -> origin/gh/fduwjj/236/orig 2025-12-04T08:57:44.1981509Z * [new branch] gh/fduwjj/237/base -> origin/gh/fduwjj/237/base 2025-12-04T08:57:44.1982576Z * [new branch] gh/fduwjj/237/head -> origin/gh/fduwjj/237/head 2025-12-04T08:57:44.1983666Z * [new branch] gh/fduwjj/237/orig -> origin/gh/fduwjj/237/orig 2025-12-04T08:57:44.1985180Z * [new branch] gh/fduwjj/238/base -> origin/gh/fduwjj/238/base 2025-12-04T08:57:44.1986334Z * [new branch] gh/fduwjj/238/head -> origin/gh/fduwjj/238/head 2025-12-04T08:57:44.1987541Z * [new branch] gh/fduwjj/238/orig -> origin/gh/fduwjj/238/orig 2025-12-04T08:57:44.1989372Z * [new branch] gh/fduwjj/239/base -> origin/gh/fduwjj/239/base 2025-12-04T08:57:44.1990626Z * [new branch] gh/fduwjj/239/head -> origin/gh/fduwjj/239/head 2025-12-04T08:57:44.1991727Z * [new branch] gh/fduwjj/239/orig -> origin/gh/fduwjj/239/orig 2025-12-04T08:57:44.1993552Z * [new branch] gh/fegin/332/base -> origin/gh/fegin/332/base 2025-12-04T08:57:44.1994679Z * [new branch] gh/fegin/332/head -> origin/gh/fegin/332/head 2025-12-04T08:57:44.1995951Z * [new branch] gh/fegin/332/orig -> origin/gh/fegin/332/orig 2025-12-04T08:57:44.1997495Z * [new branch] gh/fegin/333/base -> origin/gh/fegin/333/base 2025-12-04T08:57:44.1998650Z * [new branch] gh/fegin/333/head -> origin/gh/fegin/333/head 2025-12-04T08:57:44.1999793Z * [new branch] gh/fegin/333/orig -> origin/gh/fegin/333/orig 2025-12-04T08:57:44.2001392Z * [new branch] gh/fegin/334/base -> origin/gh/fegin/334/base 2025-12-04T08:57:44.2002509Z * [new branch] gh/fegin/334/head -> origin/gh/fegin/334/head 2025-12-04T08:57:44.2003740Z * [new branch] gh/fegin/334/orig -> origin/gh/fegin/334/orig 2025-12-04T08:57:44.2005159Z * [new branch] gh/fegin/335/base -> origin/gh/fegin/335/base 2025-12-04T08:57:44.2006285Z * [new branch] gh/fegin/335/head -> origin/gh/fegin/335/head 2025-12-04T08:57:44.2007412Z * [new branch] gh/fegin/335/orig -> origin/gh/fegin/335/orig 2025-12-04T08:57:44.2009110Z * [new branch] gh/fffrog/160/base -> origin/gh/fffrog/160/base 2025-12-04T08:57:44.2010325Z * [new branch] gh/fffrog/160/head -> origin/gh/fffrog/160/head 2025-12-04T08:57:44.2011777Z * [new branch] gh/fffrog/177/base -> origin/gh/fffrog/177/base 2025-12-04T08:57:44.2012876Z * [new branch] gh/fffrog/177/head -> origin/gh/fffrog/177/head 2025-12-04T08:57:44.2014001Z * [new branch] gh/fffrog/177/orig -> origin/gh/fffrog/177/orig 2025-12-04T08:57:44.2015425Z * [new branch] gh/fffrog/178/base -> origin/gh/fffrog/178/base 2025-12-04T08:57:44.2016523Z * [new branch] gh/fffrog/178/head -> origin/gh/fffrog/178/head 2025-12-04T08:57:44.2017648Z * [new branch] gh/fffrog/178/orig -> origin/gh/fffrog/178/orig 2025-12-04T08:57:44.2019577Z * [new branch] gh/fffrog/181/base -> origin/gh/fffrog/181/base 2025-12-04T08:57:44.2020682Z * [new branch] gh/fffrog/181/head -> origin/gh/fffrog/181/head 2025-12-04T08:57:44.2021873Z * [new branch] gh/fffrog/181/orig -> origin/gh/fffrog/181/orig 2025-12-04T08:57:44.2024174Z * [new branch] gh/fffrog/183/base -> origin/gh/fffrog/183/base 2025-12-04T08:57:44.2029247Z * [new branch] gh/fffrog/183/head -> origin/gh/fffrog/183/head 2025-12-04T08:57:44.2030618Z * [new branch] gh/fffrog/183/orig -> origin/gh/fffrog/183/orig 2025-12-04T08:57:44.2032396Z * [new branch] gh/fxdawnn/10/base -> origin/gh/fxdawnn/10/base 2025-12-04T08:57:44.2033461Z * [new branch] gh/fxdawnn/10/head -> origin/gh/fxdawnn/10/head 2025-12-04T08:57:44.2034651Z * [new branch] gh/fxdawnn/10/orig -> origin/gh/fxdawnn/10/orig 2025-12-04T08:57:44.2036333Z * [new branch] gh/fxdawnn/11/base -> origin/gh/fxdawnn/11/base 2025-12-04T08:57:44.2037565Z * [new branch] gh/fxdawnn/11/head -> origin/gh/fxdawnn/11/head 2025-12-04T08:57:44.2038597Z * [new branch] gh/fxdawnn/11/orig -> origin/gh/fxdawnn/11/orig 2025-12-04T08:57:44.2040188Z * [new branch] gh/fxdawnn/12/base -> origin/gh/fxdawnn/12/base 2025-12-04T08:57:44.2041359Z * [new branch] gh/fxdawnn/12/head -> origin/gh/fxdawnn/12/head 2025-12-04T08:57:44.2042440Z * [new branch] gh/fxdawnn/12/orig -> origin/gh/fxdawnn/12/orig 2025-12-04T08:57:44.2043914Z * [new branch] gh/fxdawnn/13/base -> origin/gh/fxdawnn/13/base 2025-12-04T08:57:44.2045283Z * [new branch] gh/fxdawnn/13/head -> origin/gh/fxdawnn/13/head 2025-12-04T08:57:44.2046227Z * [new branch] gh/fxdawnn/13/orig -> origin/gh/fxdawnn/13/orig 2025-12-04T08:57:44.2047804Z * [new branch] gh/fxdawnn/14/base -> origin/gh/fxdawnn/14/base 2025-12-04T08:57:44.2048822Z * [new branch] gh/fxdawnn/14/head -> origin/gh/fxdawnn/14/head 2025-12-04T08:57:44.2049898Z * [new branch] gh/fxdawnn/14/orig -> origin/gh/fxdawnn/14/orig 2025-12-04T08:57:44.2051296Z * [new branch] gh/fxdawnn/15/base -> origin/gh/fxdawnn/15/base 2025-12-04T08:57:44.2052387Z * [new branch] gh/fxdawnn/15/head -> origin/gh/fxdawnn/15/head 2025-12-04T08:57:44.2053492Z * [new branch] gh/fxdawnn/15/orig -> origin/gh/fxdawnn/15/orig 2025-12-04T08:57:44.2055337Z * [new branch] gh/fxdawnn/6/base -> origin/gh/fxdawnn/6/base 2025-12-04T08:57:44.2056447Z * [new branch] gh/fxdawnn/6/head -> origin/gh/fxdawnn/6/head 2025-12-04T08:57:44.2057555Z * [new branch] gh/fxdawnn/6/orig -> origin/gh/fxdawnn/6/orig 2025-12-04T08:57:44.2059026Z * [new branch] gh/fxdawnn/7/base -> origin/gh/fxdawnn/7/base 2025-12-04T08:57:44.2060284Z * [new branch] gh/fxdawnn/7/head -> origin/gh/fxdawnn/7/head 2025-12-04T08:57:44.2061335Z * [new branch] gh/fxdawnn/7/orig -> origin/gh/fxdawnn/7/orig 2025-12-04T08:57:44.2062851Z * [new branch] gh/fxdawnn/9/base -> origin/gh/fxdawnn/9/base 2025-12-04T08:57:44.2063884Z * [new branch] gh/fxdawnn/9/head -> origin/gh/fxdawnn/9/head 2025-12-04T08:57:44.2065016Z * [new branch] gh/fxdawnn/9/orig -> origin/gh/fxdawnn/9/orig 2025-12-04T08:57:44.2066749Z * [new branch] gh/galv/1/base -> origin/gh/galv/1/base 2025-12-04T08:57:44.2068126Z * [new branch] gh/galv/1/head -> origin/gh/galv/1/head 2025-12-04T08:57:44.2069294Z * [new branch] gh/galv/1/orig -> origin/gh/galv/1/orig 2025-12-04T08:57:44.2070820Z * [new branch] gh/galv/2/base -> origin/gh/galv/2/base 2025-12-04T08:57:44.2071927Z * [new branch] gh/galv/2/head -> origin/gh/galv/2/head 2025-12-04T08:57:44.2073117Z * [new branch] gh/galv/2/orig -> origin/gh/galv/2/orig 2025-12-04T08:57:44.2074730Z * [new branch] gh/galv/3/base -> origin/gh/galv/3/base 2025-12-04T08:57:44.2075868Z * [new branch] gh/galv/3/head -> origin/gh/galv/3/head 2025-12-04T08:57:44.2077256Z * [new branch] gh/galv/3/orig -> origin/gh/galv/3/orig 2025-12-04T08:57:44.2079014Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-12-04T08:57:44.2080236Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-12-04T08:57:44.2081343Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-12-04T08:57:44.2083279Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-12-04T08:57:44.2084402Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-12-04T08:57:44.2085471Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-12-04T08:57:44.2087357Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-12-04T08:57:44.2087976Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-12-04T08:57:44.2089137Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-12-04T08:57:44.2090729Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-12-04T08:57:44.2091822Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-12-04T08:57:44.2092950Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-12-04T08:57:44.2094551Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-12-04T08:57:44.2095646Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-12-04T08:57:44.2096749Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-12-04T08:57:44.2098180Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-12-04T08:57:44.2099285Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-12-04T08:57:44.2100382Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-12-04T08:57:44.2101808Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-12-04T08:57:44.2102974Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-12-04T08:57:44.2104009Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-12-04T08:57:44.2105608Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-12-04T08:57:44.2107110Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-12-04T08:57:44.2108639Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-12-04T08:57:44.2109970Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-12-04T08:57:44.2111107Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-12-04T08:57:44.2112344Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-12-04T08:57:44.2113862Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-12-04T08:57:44.2115004Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-12-04T08:57:44.2116170Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-12-04T08:57:44.2118176Z * [new branch] gh/guangyey/186/base -> origin/gh/guangyey/186/base 2025-12-04T08:57:44.2119354Z * [new branch] gh/guangyey/186/head -> origin/gh/guangyey/186/head 2025-12-04T08:57:44.2120515Z * [new branch] gh/guangyey/186/orig -> origin/gh/guangyey/186/orig 2025-12-04T08:57:44.2122131Z * [new branch] gh/guangyey/187/base -> origin/gh/guangyey/187/base 2025-12-04T08:57:44.2123340Z * [new branch] gh/guangyey/187/head -> origin/gh/guangyey/187/head 2025-12-04T08:57:44.2124813Z * [new branch] gh/guangyey/187/orig -> origin/gh/guangyey/187/orig 2025-12-04T08:57:44.2126322Z * [new branch] gh/guangyey/188/base -> origin/gh/guangyey/188/base 2025-12-04T08:57:44.2127456Z * [new branch] gh/guangyey/188/head -> origin/gh/guangyey/188/head 2025-12-04T08:57:44.2128606Z * [new branch] gh/guangyey/188/orig -> origin/gh/guangyey/188/orig 2025-12-04T08:57:44.2130112Z * [new branch] gh/guangyey/190/base -> origin/gh/guangyey/190/base 2025-12-04T08:57:44.2131250Z * [new branch] gh/guangyey/190/head -> origin/gh/guangyey/190/head 2025-12-04T08:57:44.2132424Z * [new branch] gh/guangyey/190/orig -> origin/gh/guangyey/190/orig 2025-12-04T08:57:44.2133902Z * [new branch] gh/guangyey/208/base -> origin/gh/guangyey/208/base 2025-12-04T08:57:44.2134993Z * [new branch] gh/guangyey/208/head -> origin/gh/guangyey/208/head 2025-12-04T08:57:44.2136222Z * [new branch] gh/guangyey/208/orig -> origin/gh/guangyey/208/orig 2025-12-04T08:57:44.2137760Z * [new branch] gh/guangyey/228/base -> origin/gh/guangyey/228/base 2025-12-04T08:57:44.2138851Z * [new branch] gh/guangyey/228/head -> origin/gh/guangyey/228/head 2025-12-04T08:57:44.2139955Z * [new branch] gh/guangyey/228/orig -> origin/gh/guangyey/228/orig 2025-12-04T08:57:44.2141832Z * [new branch] gh/guangyey/230/base -> origin/gh/guangyey/230/base 2025-12-04T08:57:44.2142942Z * [new branch] gh/guangyey/230/head -> origin/gh/guangyey/230/head 2025-12-04T08:57:44.2144061Z * [new branch] gh/guangyey/230/orig -> origin/gh/guangyey/230/orig 2025-12-04T08:57:44.2145614Z * [new branch] gh/guangyey/231/base -> origin/gh/guangyey/231/base 2025-12-04T08:57:44.2146683Z * [new branch] gh/guangyey/231/head -> origin/gh/guangyey/231/head 2025-12-04T08:57:44.2148076Z * [new branch] gh/guangyey/231/orig -> origin/gh/guangyey/231/orig 2025-12-04T08:57:44.2149700Z * [new branch] gh/guangyey/232/base -> origin/gh/guangyey/232/base 2025-12-04T08:57:44.2150867Z * [new branch] gh/guangyey/232/head -> origin/gh/guangyey/232/head 2025-12-04T08:57:44.2152014Z * [new branch] gh/guangyey/232/orig -> origin/gh/guangyey/232/orig 2025-12-04T08:57:44.2153666Z * [new branch] gh/guangyey/233/base -> origin/gh/guangyey/233/base 2025-12-04T08:57:44.2154800Z * [new branch] gh/guangyey/233/head -> origin/gh/guangyey/233/head 2025-12-04T08:57:44.2155929Z * [new branch] gh/guangyey/233/orig -> origin/gh/guangyey/233/orig 2025-12-04T08:57:44.2157507Z * [new branch] gh/guangyey/234/base -> origin/gh/guangyey/234/base 2025-12-04T08:57:44.2158635Z * [new branch] gh/guangyey/234/head -> origin/gh/guangyey/234/head 2025-12-04T08:57:44.2159876Z * [new branch] gh/guangyey/234/orig -> origin/gh/guangyey/234/orig 2025-12-04T08:57:44.2161485Z * [new branch] gh/guangyey/235/base -> origin/gh/guangyey/235/base 2025-12-04T08:57:44.2162555Z * [new branch] gh/guangyey/235/head -> origin/gh/guangyey/235/head 2025-12-04T08:57:44.2163645Z * [new branch] gh/guangyey/235/orig -> origin/gh/guangyey/235/orig 2025-12-04T08:57:44.2165159Z * [new branch] gh/guangyey/236/base -> origin/gh/guangyey/236/base 2025-12-04T08:57:44.2166268Z * [new branch] gh/guangyey/236/head -> origin/gh/guangyey/236/head 2025-12-04T08:57:44.2167367Z * [new branch] gh/guangyey/236/orig -> origin/gh/guangyey/236/orig 2025-12-04T08:57:44.2169091Z * [new branch] gh/guangyey/237/base -> origin/gh/guangyey/237/base 2025-12-04T08:57:44.2170127Z * [new branch] gh/guangyey/237/head -> origin/gh/guangyey/237/head 2025-12-04T08:57:44.2171170Z * [new branch] gh/guangyey/237/orig -> origin/gh/guangyey/237/orig 2025-12-04T08:57:44.2172677Z * [new branch] gh/guangyey/238/base -> origin/gh/guangyey/238/base 2025-12-04T08:57:44.2173783Z * [new branch] gh/guangyey/238/head -> origin/gh/guangyey/238/head 2025-12-04T08:57:44.2175288Z * [new branch] gh/guangyey/239/base -> origin/gh/guangyey/239/base 2025-12-04T08:57:44.2176407Z * [new branch] gh/guangyey/239/head -> origin/gh/guangyey/239/head 2025-12-04T08:57:44.2177495Z * [new branch] gh/guangyey/239/orig -> origin/gh/guangyey/239/orig 2025-12-04T08:57:44.2179032Z * [new branch] gh/guangyey/240/base -> origin/gh/guangyey/240/base 2025-12-04T08:57:44.2180097Z * [new branch] gh/guangyey/240/head -> origin/gh/guangyey/240/head 2025-12-04T08:57:44.2181214Z * [new branch] gh/guangyey/240/orig -> origin/gh/guangyey/240/orig 2025-12-04T08:57:44.2182765Z * [new branch] gh/guangyey/241/base -> origin/gh/guangyey/241/base 2025-12-04T08:57:44.2183870Z * [new branch] gh/guangyey/241/head -> origin/gh/guangyey/241/head 2025-12-04T08:57:44.2184983Z * [new branch] gh/guangyey/241/orig -> origin/gh/guangyey/241/orig 2025-12-04T08:57:44.2186461Z * [new branch] gh/guangyey/242/base -> origin/gh/guangyey/242/base 2025-12-04T08:57:44.2187613Z * [new branch] gh/guangyey/242/head -> origin/gh/guangyey/242/head 2025-12-04T08:57:44.2189009Z * [new branch] gh/guangyey/242/orig -> origin/gh/guangyey/242/orig 2025-12-04T08:57:44.2190554Z * [new branch] gh/guangyey/243/base -> origin/gh/guangyey/243/base 2025-12-04T08:57:44.2191649Z * [new branch] gh/guangyey/243/head -> origin/gh/guangyey/243/head 2025-12-04T08:57:44.2192794Z * [new branch] gh/guangyey/243/orig -> origin/gh/guangyey/243/orig 2025-12-04T08:57:44.2194406Z * [new branch] gh/guangyey/244/base -> origin/gh/guangyey/244/base 2025-12-04T08:57:44.2195498Z * [new branch] gh/guangyey/244/head -> origin/gh/guangyey/244/head 2025-12-04T08:57:44.2196798Z * [new branch] gh/guangyey/244/orig -> origin/gh/guangyey/244/orig 2025-12-04T08:57:44.2198403Z * [new branch] gh/guangyey/245/base -> origin/gh/guangyey/245/base 2025-12-04T08:57:44.2199552Z * [new branch] gh/guangyey/245/head -> origin/gh/guangyey/245/head 2025-12-04T08:57:44.2200742Z * [new branch] gh/guangyey/245/orig -> origin/gh/guangyey/245/orig 2025-12-04T08:57:44.2202268Z * [new branch] gh/guangyey/246/base -> origin/gh/guangyey/246/base 2025-12-04T08:57:44.2203392Z * [new branch] gh/guangyey/246/head -> origin/gh/guangyey/246/head 2025-12-04T08:57:44.2204497Z * [new branch] gh/guangyey/246/orig -> origin/gh/guangyey/246/orig 2025-12-04T08:57:44.2206011Z * [new branch] gh/guangyey/247/base -> origin/gh/guangyey/247/base 2025-12-04T08:57:44.2207095Z * [new branch] gh/guangyey/247/head -> origin/gh/guangyey/247/head 2025-12-04T08:57:44.2208239Z * [new branch] gh/guangyey/247/orig -> origin/gh/guangyey/247/orig 2025-12-04T08:57:44.2209722Z * [new branch] gh/guangyey/248/base -> origin/gh/guangyey/248/base 2025-12-04T08:57:44.2210771Z * [new branch] gh/guangyey/248/head -> origin/gh/guangyey/248/head 2025-12-04T08:57:44.2211863Z * [new branch] gh/guangyey/248/orig -> origin/gh/guangyey/248/orig 2025-12-04T08:57:44.2213585Z * [new branch] gh/guangyey/249/base -> origin/gh/guangyey/249/base 2025-12-04T08:57:44.2214475Z * [new branch] gh/guangyey/249/head -> origin/gh/guangyey/249/head 2025-12-04T08:57:44.2215663Z * [new branch] gh/guangyey/249/orig -> origin/gh/guangyey/249/orig 2025-12-04T08:57:44.2217185Z * [new branch] gh/guangyey/250/base -> origin/gh/guangyey/250/base 2025-12-04T08:57:44.2218297Z * [new branch] gh/guangyey/250/head -> origin/gh/guangyey/250/head 2025-12-04T08:57:44.2219384Z * [new branch] gh/guangyey/250/orig -> origin/gh/guangyey/250/orig 2025-12-04T08:57:44.2220840Z * [new branch] gh/guangyey/251/base -> origin/gh/guangyey/251/base 2025-12-04T08:57:44.2221963Z * [new branch] gh/guangyey/251/head -> origin/gh/guangyey/251/head 2025-12-04T08:57:44.2223059Z * [new branch] gh/guangyey/251/orig -> origin/gh/guangyey/251/orig 2025-12-04T08:57:44.2225200Z * [new branch] gh/guangyey/252/base -> origin/gh/guangyey/252/base 2025-12-04T08:57:44.2226812Z * [new branch] gh/guangyey/252/head -> origin/gh/guangyey/252/head 2025-12-04T08:57:44.2228129Z * [new branch] gh/guangyey/252/orig -> origin/gh/guangyey/252/orig 2025-12-04T08:57:44.2229788Z * [new branch] gh/guangyey/253/base -> origin/gh/guangyey/253/base 2025-12-04T08:57:44.2230932Z * [new branch] gh/guangyey/253/head -> origin/gh/guangyey/253/head 2025-12-04T08:57:44.2232055Z * [new branch] gh/guangyey/253/orig -> origin/gh/guangyey/253/orig 2025-12-04T08:57:44.2233583Z * [new branch] gh/guangyey/254/base -> origin/gh/guangyey/254/base 2025-12-04T08:57:44.2234714Z * [new branch] gh/guangyey/254/head -> origin/gh/guangyey/254/head 2025-12-04T08:57:44.2236337Z * [new branch] gh/guangyey/254/orig -> origin/gh/guangyey/254/orig 2025-12-04T08:57:44.2237940Z * [new branch] gh/guangyey/255/base -> origin/gh/guangyey/255/base 2025-12-04T08:57:44.2239167Z * [new branch] gh/guangyey/255/head -> origin/gh/guangyey/255/head 2025-12-04T08:57:44.2240304Z * [new branch] gh/guangyey/255/orig -> origin/gh/guangyey/255/orig 2025-12-04T08:57:44.2242207Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-12-04T08:57:44.2243741Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-12-04T08:57:44.2244861Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-12-04T08:57:44.2246340Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-12-04T08:57:44.2247625Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-12-04T08:57:44.2248659Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-12-04T08:57:44.2250057Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-12-04T08:57:44.2251131Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-12-04T08:57:44.2253918Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-12-04T08:57:44.2255202Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-12-04T08:57:44.2256271Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-12-04T08:57:44.2257383Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-12-04T08:57:44.2259323Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-12-04T08:57:44.2260545Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-12-04T08:57:44.2261479Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-12-04T08:57:44.2264079Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-12-04T08:57:44.2264371Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-12-04T08:57:44.2265395Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-12-04T08:57:44.2266865Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-12-04T08:57:44.2268279Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-12-04T08:57:44.2269568Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-12-04T08:57:44.2270930Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-12-04T08:57:44.2272055Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-12-04T08:57:44.2273181Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-12-04T08:57:44.2274704Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-12-04T08:57:44.2275846Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-12-04T08:57:44.2277143Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-12-04T08:57:44.2278717Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-12-04T08:57:44.2279989Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-12-04T08:57:44.2281108Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-12-04T08:57:44.2282560Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-12-04T08:57:44.2283643Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-12-04T08:57:44.2284740Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-12-04T08:57:44.2286204Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-12-04T08:57:44.2287276Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-12-04T08:57:44.2288351Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-12-04T08:57:44.2289798Z * [new branch] gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base 2025-12-04T08:57:44.2290886Z * [new branch] gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head 2025-12-04T08:57:44.2291972Z * [new branch] gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig 2025-12-04T08:57:44.2293698Z * [new branch] gh/guilhermeleobas/247/base -> origin/gh/guilhermeleobas/247/base 2025-12-04T08:57:44.2294791Z * [new branch] gh/guilhermeleobas/247/head -> origin/gh/guilhermeleobas/247/head 2025-12-04T08:57:44.2295936Z * [new branch] gh/guilhermeleobas/247/orig -> origin/gh/guilhermeleobas/247/orig 2025-12-04T08:57:44.2297447Z * [new branch] gh/guilhermeleobas/248/base -> origin/gh/guilhermeleobas/248/base 2025-12-04T08:57:44.2298564Z * [new branch] gh/guilhermeleobas/248/head -> origin/gh/guilhermeleobas/248/head 2025-12-04T08:57:44.2299693Z * [new branch] gh/guilhermeleobas/248/orig -> origin/gh/guilhermeleobas/248/orig 2025-12-04T08:57:44.2301147Z * [new branch] gh/guilhermeleobas/250/base -> origin/gh/guilhermeleobas/250/base 2025-12-04T08:57:44.2302325Z * [new branch] gh/guilhermeleobas/250/head -> origin/gh/guilhermeleobas/250/head 2025-12-04T08:57:44.2303402Z * [new branch] gh/guilhermeleobas/250/orig -> origin/gh/guilhermeleobas/250/orig 2025-12-04T08:57:44.2305268Z * [new branch] gh/guilhermeleobas/253/base -> origin/gh/guilhermeleobas/253/base 2025-12-04T08:57:44.2306382Z * [new branch] gh/guilhermeleobas/253/head -> origin/gh/guilhermeleobas/253/head 2025-12-04T08:57:44.2307610Z * [new branch] gh/guilhermeleobas/253/orig -> origin/gh/guilhermeleobas/253/orig 2025-12-04T08:57:44.2309508Z * [new branch] gh/guilhermeleobas/254/base -> origin/gh/guilhermeleobas/254/base 2025-12-04T08:57:44.2310657Z * [new branch] gh/guilhermeleobas/254/head -> origin/gh/guilhermeleobas/254/head 2025-12-04T08:57:44.2311800Z * [new branch] gh/guilhermeleobas/254/orig -> origin/gh/guilhermeleobas/254/orig 2025-12-04T08:57:44.2313810Z * [new branch] gh/guilhermeleobas/255/base -> origin/gh/guilhermeleobas/255/base 2025-12-04T08:57:44.2314950Z * [new branch] gh/guilhermeleobas/255/head -> origin/gh/guilhermeleobas/255/head 2025-12-04T08:57:44.2316092Z * [new branch] gh/guilhermeleobas/255/orig -> origin/gh/guilhermeleobas/255/orig 2025-12-04T08:57:44.2317724Z * [new branch] gh/guilhermeleobas/256/base -> origin/gh/guilhermeleobas/256/base 2025-12-04T08:57:44.2318876Z * [new branch] gh/guilhermeleobas/256/head -> origin/gh/guilhermeleobas/256/head 2025-12-04T08:57:44.2320308Z * [new branch] gh/guilhermeleobas/256/orig -> origin/gh/guilhermeleobas/256/orig 2025-12-04T08:57:44.2321715Z * [new branch] gh/guilhermeleobas/257/base -> origin/gh/guilhermeleobas/257/base 2025-12-04T08:57:44.2323308Z * [new branch] gh/guilhermeleobas/257/head -> origin/gh/guilhermeleobas/257/head 2025-12-04T08:57:44.2325116Z * [new branch] gh/guilhermeleobas/257/orig -> origin/gh/guilhermeleobas/257/orig 2025-12-04T08:57:44.2326669Z * [new branch] gh/guilhermeleobas/258/base -> origin/gh/guilhermeleobas/258/base 2025-12-04T08:57:44.2328176Z * [new branch] gh/guilhermeleobas/258/head -> origin/gh/guilhermeleobas/258/head 2025-12-04T08:57:44.2328933Z * [new branch] gh/guilhermeleobas/258/orig -> origin/gh/guilhermeleobas/258/orig 2025-12-04T08:57:44.2330507Z * [new branch] gh/guilhermeleobas/259/base -> origin/gh/guilhermeleobas/259/base 2025-12-04T08:57:44.2333733Z * [new branch] gh/guilhermeleobas/259/head -> origin/gh/guilhermeleobas/259/head 2025-12-04T08:57:44.2335093Z * [new branch] gh/guilhermeleobas/259/orig -> origin/gh/guilhermeleobas/259/orig 2025-12-04T08:57:44.2335400Z * [new branch] gh/guilhermeleobas/260/base -> origin/gh/guilhermeleobas/260/base 2025-12-04T08:57:44.2335897Z * [new branch] gh/guilhermeleobas/260/head -> origin/gh/guilhermeleobas/260/head 2025-12-04T08:57:44.2337230Z * [new branch] gh/guilhermeleobas/260/orig -> origin/gh/guilhermeleobas/260/orig 2025-12-04T08:57:44.2338651Z * [new branch] gh/guilhermeleobas/261/base -> origin/gh/guilhermeleobas/261/base 2025-12-04T08:57:44.2339722Z * [new branch] gh/guilhermeleobas/261/head -> origin/gh/guilhermeleobas/261/head 2025-12-04T08:57:44.2340829Z * [new branch] gh/guilhermeleobas/261/orig -> origin/gh/guilhermeleobas/261/orig 2025-12-04T08:57:44.2342442Z * [new branch] gh/guilhermeleobas/262/base -> origin/gh/guilhermeleobas/262/base 2025-12-04T08:57:44.2343643Z * [new branch] gh/guilhermeleobas/262/head -> origin/gh/guilhermeleobas/262/head 2025-12-04T08:57:44.2344685Z * [new branch] gh/guilhermeleobas/262/orig -> origin/gh/guilhermeleobas/262/orig 2025-12-04T08:57:44.2346218Z * [new branch] gh/guilhermeleobas/263/base -> origin/gh/guilhermeleobas/263/base 2025-12-04T08:57:44.2347471Z * [new branch] gh/guilhermeleobas/263/head -> origin/gh/guilhermeleobas/263/head 2025-12-04T08:57:44.2348787Z * [new branch] gh/guilhermeleobas/263/orig -> origin/gh/guilhermeleobas/263/orig 2025-12-04T08:57:44.2350412Z * [new branch] gh/guilhermeleobas/264/base -> origin/gh/guilhermeleobas/264/base 2025-12-04T08:57:44.2351514Z * [new branch] gh/guilhermeleobas/264/head -> origin/gh/guilhermeleobas/264/head 2025-12-04T08:57:44.2352667Z * [new branch] gh/guilhermeleobas/264/orig -> origin/gh/guilhermeleobas/264/orig 2025-12-04T08:57:44.2354170Z * [new branch] gh/guilhermeleobas/265/base -> origin/gh/guilhermeleobas/265/base 2025-12-04T08:57:44.2355296Z * [new branch] gh/guilhermeleobas/265/head -> origin/gh/guilhermeleobas/265/head 2025-12-04T08:57:44.2356422Z * [new branch] gh/guilhermeleobas/265/orig -> origin/gh/guilhermeleobas/265/orig 2025-12-04T08:57:44.2358128Z * [new branch] gh/guilhermeleobas/266/base -> origin/gh/guilhermeleobas/266/base 2025-12-04T08:57:44.2359267Z * [new branch] gh/guilhermeleobas/266/head -> origin/gh/guilhermeleobas/266/head 2025-12-04T08:57:44.2360451Z * [new branch] gh/guilhermeleobas/266/orig -> origin/gh/guilhermeleobas/266/orig 2025-12-04T08:57:44.2362262Z * [new branch] gh/guilhermeleobas/267/base -> origin/gh/guilhermeleobas/267/base 2025-12-04T08:57:44.2363351Z * [new branch] gh/guilhermeleobas/267/head -> origin/gh/guilhermeleobas/267/head 2025-12-04T08:57:44.2364484Z * [new branch] gh/guilhermeleobas/267/orig -> origin/gh/guilhermeleobas/267/orig 2025-12-04T08:57:44.2366286Z * [new branch] gh/hameerabbasi/1/base -> origin/gh/hameerabbasi/1/base 2025-12-04T08:57:44.2367408Z * [new branch] gh/hameerabbasi/1/head -> origin/gh/hameerabbasi/1/head 2025-12-04T08:57:44.2368849Z * [new branch] gh/hameerabbasi/2/base -> origin/gh/hameerabbasi/2/base 2025-12-04T08:57:44.2369926Z * [new branch] gh/hameerabbasi/2/head -> origin/gh/hameerabbasi/2/head 2025-12-04T08:57:44.2371068Z * [new branch] gh/hameerabbasi/2/orig -> origin/gh/hameerabbasi/2/orig 2025-12-04T08:57:44.2372572Z * [new branch] gh/hameerabbasi/3/base -> origin/gh/hameerabbasi/3/base 2025-12-04T08:57:44.2373709Z * [new branch] gh/hameerabbasi/3/head -> origin/gh/hameerabbasi/3/head 2025-12-04T08:57:44.2374956Z * [new branch] gh/hameerabbasi/3/orig -> origin/gh/hameerabbasi/3/orig 2025-12-04T08:57:44.2376342Z * [new branch] gh/hameerabbasi/4/base -> origin/gh/hameerabbasi/4/base 2025-12-04T08:57:44.2377436Z * [new branch] gh/hameerabbasi/4/head -> origin/gh/hameerabbasi/4/head 2025-12-04T08:57:44.2378863Z * [new branch] gh/hameerabbasi/4/orig -> origin/gh/hameerabbasi/4/orig 2025-12-04T08:57:44.2380273Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-12-04T08:57:44.2381526Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-12-04T08:57:44.2382951Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-12-04T08:57:44.2384488Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-12-04T08:57:44.2385923Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-12-04T08:57:44.2387442Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-12-04T08:57:44.2389458Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-12-04T08:57:44.2390696Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-12-04T08:57:44.2392503Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-12-04T08:57:44.2393725Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-12-04T08:57:44.2395147Z * [new branch] gh/isuruf/146/base -> origin/gh/isuruf/146/base 2025-12-04T08:57:44.2396296Z * [new branch] gh/isuruf/146/head -> origin/gh/isuruf/146/head 2025-12-04T08:57:44.2397489Z * [new branch] gh/isuruf/146/orig -> origin/gh/isuruf/146/orig 2025-12-04T08:57:44.2398989Z * [new branch] gh/isuruf/158/base -> origin/gh/isuruf/158/base 2025-12-04T08:57:44.2400202Z * [new branch] gh/isuruf/158/head -> origin/gh/isuruf/158/head 2025-12-04T08:57:44.2401977Z * [new branch] gh/isuruf/159/base -> origin/gh/isuruf/159/base 2025-12-04T08:57:44.2403041Z * [new branch] gh/isuruf/159/head -> origin/gh/isuruf/159/head 2025-12-04T08:57:44.2404489Z * [new branch] gh/isuruf/160/base -> origin/gh/isuruf/160/base 2025-12-04T08:57:44.2405638Z * [new branch] gh/isuruf/160/head -> origin/gh/isuruf/160/head 2025-12-04T08:57:44.2406755Z * [new branch] gh/isuruf/160/orig -> origin/gh/isuruf/160/orig 2025-12-04T08:57:44.2408202Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-12-04T08:57:44.2409291Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-12-04T08:57:44.2410386Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-12-04T08:57:44.2412212Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-12-04T08:57:44.2413331Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-12-04T08:57:44.2414401Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-12-04T08:57:44.2415882Z * [new branch] gh/jamesjwu/187/base -> origin/gh/jamesjwu/187/base 2025-12-04T08:57:44.2416894Z * [new branch] gh/jamesjwu/187/head -> origin/gh/jamesjwu/187/head 2025-12-04T08:57:44.2417971Z * [new branch] gh/jamesjwu/187/orig -> origin/gh/jamesjwu/187/orig 2025-12-04T08:57:44.2419434Z * [new branch] gh/jamesjwu/196/base -> origin/gh/jamesjwu/196/base 2025-12-04T08:57:44.2420507Z * [new branch] gh/jamesjwu/196/head -> origin/gh/jamesjwu/196/head 2025-12-04T08:57:44.2421593Z * [new branch] gh/jamesjwu/196/orig -> origin/gh/jamesjwu/196/orig 2025-12-04T08:57:44.2422974Z * [new branch] gh/jamesjwu/198/base -> origin/gh/jamesjwu/198/base 2025-12-04T08:57:44.2424478Z * [new branch] gh/jamesjwu/198/head -> origin/gh/jamesjwu/198/head 2025-12-04T08:57:44.2425643Z * [new branch] gh/jamesjwu/198/orig -> origin/gh/jamesjwu/198/orig 2025-12-04T08:57:44.2427389Z * [new branch] gh/jamesjwu/207/base -> origin/gh/jamesjwu/207/base 2025-12-04T08:57:44.2428823Z * [new branch] gh/jamesjwu/207/head -> origin/gh/jamesjwu/207/head 2025-12-04T08:57:44.2430151Z * [new branch] gh/jamesjwu/207/orig -> origin/gh/jamesjwu/207/orig 2025-12-04T08:57:44.2431796Z * [new branch] gh/jamesjwu/208/base -> origin/gh/jamesjwu/208/base 2025-12-04T08:57:44.2432931Z * [new branch] gh/jamesjwu/208/head -> origin/gh/jamesjwu/208/head 2025-12-04T08:57:44.2434080Z * [new branch] gh/jamesjwu/208/orig -> origin/gh/jamesjwu/208/orig 2025-12-04T08:57:44.2435693Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-12-04T08:57:44.2436949Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-12-04T08:57:44.2438269Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-12-04T08:57:44.2439291Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-12-04T08:57:44.2441283Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-12-04T08:57:44.2442345Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-12-04T08:57:44.2443626Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-12-04T08:57:44.2444817Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-12-04T08:57:44.2446069Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-12-04T08:57:44.2447148Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-12-04T08:57:44.2448451Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-12-04T08:57:44.2449525Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-12-04T08:57:44.2450857Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-12-04T08:57:44.2451892Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-12-04T08:57:44.2453295Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-12-04T08:57:44.2454340Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-12-04T08:57:44.2456149Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-12-04T08:57:44.2457241Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-12-04T08:57:44.2458593Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-12-04T08:57:44.2459565Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-12-04T08:57:44.2480780Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-12-04T08:57:44.2481204Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-12-04T08:57:44.2481464Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-12-04T08:57:44.2481701Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-12-04T08:57:44.2481952Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-12-04T08:57:44.2482187Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-12-04T08:57:44.2482433Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-12-04T08:57:44.2482663Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-12-04T08:57:44.2482899Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-12-04T08:57:44.2483151Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-12-04T08:57:44.2483388Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-12-04T08:57:44.2483629Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-12-04T08:57:44.2483879Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-12-04T08:57:44.2484110Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-12-04T08:57:44.2484349Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-12-04T08:57:44.2484591Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-12-04T08:57:44.2484821Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-12-04T08:57:44.2485066Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-12-04T08:57:44.2485300Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-12-04T08:57:44.2486058Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-12-04T08:57:44.2487813Z * [new branch] gh/janeyx99/302/base -> origin/gh/janeyx99/302/base 2025-12-04T08:57:44.2489004Z * [new branch] gh/janeyx99/302/head -> origin/gh/janeyx99/302/head 2025-12-04T08:57:44.2490227Z * [new branch] gh/janeyx99/303/base -> origin/gh/janeyx99/303/base 2025-12-04T08:57:44.2491270Z * [new branch] gh/janeyx99/303/head -> origin/gh/janeyx99/303/head 2025-12-04T08:57:44.2492736Z * [new branch] gh/janeyx99/305/base -> origin/gh/janeyx99/305/base 2025-12-04T08:57:44.2493851Z * [new branch] gh/janeyx99/305/head -> origin/gh/janeyx99/305/head 2025-12-04T08:57:44.2495162Z * [new branch] gh/janeyx99/306/base -> origin/gh/janeyx99/306/base 2025-12-04T08:57:44.2496363Z * [new branch] gh/janeyx99/306/head -> origin/gh/janeyx99/306/head 2025-12-04T08:57:44.2497952Z * [new branch] gh/janeyx99/314/base -> origin/gh/janeyx99/314/base 2025-12-04T08:57:44.2499085Z * [new branch] gh/janeyx99/314/head -> origin/gh/janeyx99/314/head 2025-12-04T08:57:44.2500187Z * [new branch] gh/janeyx99/314/orig -> origin/gh/janeyx99/314/orig 2025-12-04T08:57:44.2501688Z * [new branch] gh/janeyx99/315/base -> origin/gh/janeyx99/315/base 2025-12-04T08:57:44.2502802Z * [new branch] gh/janeyx99/315/head -> origin/gh/janeyx99/315/head 2025-12-04T08:57:44.2503880Z * [new branch] gh/janeyx99/315/orig -> origin/gh/janeyx99/315/orig 2025-12-04T08:57:44.2505385Z * [new branch] gh/janeyx99/316/base -> origin/gh/janeyx99/316/base 2025-12-04T08:57:44.2506528Z * [new branch] gh/janeyx99/316/head -> origin/gh/janeyx99/316/head 2025-12-04T08:57:44.2507884Z * [new branch] gh/janeyx99/316/orig -> origin/gh/janeyx99/316/orig 2025-12-04T08:57:44.2509611Z * [new branch] gh/janeyx99/317/base -> origin/gh/janeyx99/317/base 2025-12-04T08:57:44.2510693Z * [new branch] gh/janeyx99/317/head -> origin/gh/janeyx99/317/head 2025-12-04T08:57:44.2511825Z * [new branch] gh/janeyx99/317/orig -> origin/gh/janeyx99/317/orig 2025-12-04T08:57:44.2513522Z * [new branch] gh/janeyx99/325/base -> origin/gh/janeyx99/325/base 2025-12-04T08:57:44.2514636Z * [new branch] gh/janeyx99/325/head -> origin/gh/janeyx99/325/head 2025-12-04T08:57:44.2515768Z * [new branch] gh/janeyx99/325/orig -> origin/gh/janeyx99/325/orig 2025-12-04T08:57:44.2517299Z * [new branch] gh/janeyx99/327/base -> origin/gh/janeyx99/327/base 2025-12-04T08:57:44.2518412Z * [new branch] gh/janeyx99/327/head -> origin/gh/janeyx99/327/head 2025-12-04T08:57:44.2519665Z * [new branch] gh/janeyx99/327/orig -> origin/gh/janeyx99/327/orig 2025-12-04T08:57:44.2521182Z * [new branch] gh/janeyx99/328/base -> origin/gh/janeyx99/328/base 2025-12-04T08:57:44.2522409Z * [new branch] gh/janeyx99/328/head -> origin/gh/janeyx99/328/head 2025-12-04T08:57:44.2523462Z * [new branch] gh/janeyx99/328/orig -> origin/gh/janeyx99/328/orig 2025-12-04T08:57:44.2526093Z * [new branch] gh/janeyx99/329/base -> origin/gh/janeyx99/329/base 2025-12-04T08:57:44.2527332Z * [new branch] gh/janeyx99/329/head -> origin/gh/janeyx99/329/head 2025-12-04T08:57:44.2528448Z * [new branch] gh/janeyx99/329/orig -> origin/gh/janeyx99/329/orig 2025-12-04T08:57:44.2530584Z * [new branch] gh/janeyx99/330/base -> origin/gh/janeyx99/330/base 2025-12-04T08:57:44.2531711Z * [new branch] gh/janeyx99/330/head -> origin/gh/janeyx99/330/head 2025-12-04T08:57:44.2533048Z * [new branch] gh/janeyx99/330/orig -> origin/gh/janeyx99/330/orig 2025-12-04T08:57:44.2534465Z * [new branch] gh/janeyx99/331/base -> origin/gh/janeyx99/331/base 2025-12-04T08:57:44.2536052Z * [new branch] gh/janeyx99/331/head -> origin/gh/janeyx99/331/head 2025-12-04T08:57:44.2537437Z * [new branch] gh/janeyx99/331/orig -> origin/gh/janeyx99/331/orig 2025-12-04T08:57:44.2538763Z * [new branch] gh/janeyx99/332/base -> origin/gh/janeyx99/332/base 2025-12-04T08:57:44.2539857Z * [new branch] gh/janeyx99/332/head -> origin/gh/janeyx99/332/head 2025-12-04T08:57:44.2540926Z * [new branch] gh/janeyx99/332/orig -> origin/gh/janeyx99/332/orig 2025-12-04T08:57:44.2542358Z * [new branch] gh/janeyx99/333/base -> origin/gh/janeyx99/333/base 2025-12-04T08:57:44.2543437Z * [new branch] gh/janeyx99/333/head -> origin/gh/janeyx99/333/head 2025-12-04T08:57:44.2544507Z * [new branch] gh/janeyx99/333/orig -> origin/gh/janeyx99/333/orig 2025-12-04T08:57:44.2546200Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-12-04T08:57:44.2547365Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-12-04T08:57:44.2548778Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-12-04T08:57:44.2550788Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-12-04T08:57:44.2551664Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-12-04T08:57:44.2553182Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-12-04T08:57:44.2554322Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-12-04T08:57:44.2555516Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-12-04T08:57:44.2556950Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-12-04T08:57:44.2558088Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-12-04T08:57:44.2559173Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-12-04T08:57:44.2560868Z * [new branch] gh/jansel/533/base -> origin/gh/jansel/533/base 2025-12-04T08:57:44.2561965Z * [new branch] gh/jansel/533/head -> origin/gh/jansel/533/head 2025-12-04T08:57:44.2563190Z * [new branch] gh/jansel/533/orig -> origin/gh/jansel/533/orig 2025-12-04T08:57:44.2564641Z * [new branch] gh/jansel/552/base -> origin/gh/jansel/552/base 2025-12-04T08:57:44.2565679Z * [new branch] gh/jansel/552/head -> origin/gh/jansel/552/head 2025-12-04T08:57:44.2566784Z * [new branch] gh/jansel/552/orig -> origin/gh/jansel/552/orig 2025-12-04T08:57:44.2568270Z * [new branch] gh/jansel/553/base -> origin/gh/jansel/553/base 2025-12-04T08:57:44.2569355Z * [new branch] gh/jansel/553/head -> origin/gh/jansel/553/head 2025-12-04T08:57:44.2570444Z * [new branch] gh/jansel/553/orig -> origin/gh/jansel/553/orig 2025-12-04T08:57:44.2571917Z * [new branch] gh/jansel/554/base -> origin/gh/jansel/554/base 2025-12-04T08:57:44.2573006Z * [new branch] gh/jansel/554/head -> origin/gh/jansel/554/head 2025-12-04T08:57:44.2574075Z * [new branch] gh/jansel/554/orig -> origin/gh/jansel/554/orig 2025-12-04T08:57:44.2575646Z * [new branch] gh/jansel/555/base -> origin/gh/jansel/555/base 2025-12-04T08:57:44.2576749Z * [new branch] gh/jansel/555/head -> origin/gh/jansel/555/head 2025-12-04T08:57:44.2577936Z * [new branch] gh/jansel/555/orig -> origin/gh/jansel/555/orig 2025-12-04T08:57:44.2579307Z * [new branch] gh/jansel/556/base -> origin/gh/jansel/556/base 2025-12-04T08:57:44.2580411Z * [new branch] gh/jansel/556/head -> origin/gh/jansel/556/head 2025-12-04T08:57:44.2581491Z * [new branch] gh/jansel/556/orig -> origin/gh/jansel/556/orig 2025-12-04T08:57:44.2582929Z * [new branch] gh/jansel/557/base -> origin/gh/jansel/557/base 2025-12-04T08:57:44.2584011Z * [new branch] gh/jansel/557/head -> origin/gh/jansel/557/head 2025-12-04T08:57:44.2585217Z * [new branch] gh/jansel/557/orig -> origin/gh/jansel/557/orig 2025-12-04T08:57:44.2586510Z * [new branch] gh/jansel/558/base -> origin/gh/jansel/558/base 2025-12-04T08:57:44.2587863Z * [new branch] gh/jansel/558/head -> origin/gh/jansel/558/head 2025-12-04T08:57:44.2589065Z * [new branch] gh/jansel/558/orig -> origin/gh/jansel/558/orig 2025-12-04T08:57:44.2590668Z * [new branch] gh/jansel/559/base -> origin/gh/jansel/559/base 2025-12-04T08:57:44.2591741Z * [new branch] gh/jansel/559/head -> origin/gh/jansel/559/head 2025-12-04T08:57:44.2592850Z * [new branch] gh/jansel/559/orig -> origin/gh/jansel/559/orig 2025-12-04T08:57:44.2594305Z * [new branch] gh/jansel/560/base -> origin/gh/jansel/560/base 2025-12-04T08:57:44.2595425Z * [new branch] gh/jansel/560/head -> origin/gh/jansel/560/head 2025-12-04T08:57:44.2596556Z * [new branch] gh/jansel/560/orig -> origin/gh/jansel/560/orig 2025-12-04T08:57:44.2598044Z * [new branch] gh/jansel/561/base -> origin/gh/jansel/561/base 2025-12-04T08:57:44.2599181Z * [new branch] gh/jansel/561/head -> origin/gh/jansel/561/head 2025-12-04T08:57:44.2600396Z * [new branch] gh/jansel/561/orig -> origin/gh/jansel/561/orig 2025-12-04T08:57:44.2601817Z * [new branch] gh/jansel/562/base -> origin/gh/jansel/562/base 2025-12-04T08:57:44.2602906Z * [new branch] gh/jansel/562/head -> origin/gh/jansel/562/head 2025-12-04T08:57:44.2603979Z * [new branch] gh/jansel/562/orig -> origin/gh/jansel/562/orig 2025-12-04T08:57:44.2605510Z * [new branch] gh/jansel/563/base -> origin/gh/jansel/563/base 2025-12-04T08:57:44.2606651Z * [new branch] gh/jansel/563/head -> origin/gh/jansel/563/head 2025-12-04T08:57:44.2607752Z * [new branch] gh/jansel/563/orig -> origin/gh/jansel/563/orig 2025-12-04T08:57:44.2609608Z * [new branch] gh/jansel/564/base -> origin/gh/jansel/564/base 2025-12-04T08:57:44.2611001Z * [new branch] gh/jansel/564/head -> origin/gh/jansel/564/head 2025-12-04T08:57:44.2611820Z * [new branch] gh/jansel/564/orig -> origin/gh/jansel/564/orig 2025-12-04T08:57:44.2613336Z * [new branch] gh/jansel/565/base -> origin/gh/jansel/565/base 2025-12-04T08:57:44.2614435Z * [new branch] gh/jansel/565/head -> origin/gh/jansel/565/head 2025-12-04T08:57:44.2615492Z * [new branch] gh/jansel/565/orig -> origin/gh/jansel/565/orig 2025-12-04T08:57:44.2617038Z * [new branch] gh/jansel/566/base -> origin/gh/jansel/566/base 2025-12-04T08:57:44.2618093Z * [new branch] gh/jansel/566/head -> origin/gh/jansel/566/head 2025-12-04T08:57:44.2619203Z * [new branch] gh/jansel/566/orig -> origin/gh/jansel/566/orig 2025-12-04T08:57:44.2620765Z * [new branch] gh/jansel/567/base -> origin/gh/jansel/567/base 2025-12-04T08:57:44.2621824Z * [new branch] gh/jansel/567/head -> origin/gh/jansel/567/head 2025-12-04T08:57:44.2622995Z * [new branch] gh/jansel/567/orig -> origin/gh/jansel/567/orig 2025-12-04T08:57:44.2624968Z * [new branch] gh/jansel/568/base -> origin/gh/jansel/568/base 2025-12-04T08:57:44.2626090Z * [new branch] gh/jansel/568/head -> origin/gh/jansel/568/head 2025-12-04T08:57:44.2627430Z * [new branch] gh/jansel/568/orig -> origin/gh/jansel/568/orig 2025-12-04T08:57:44.2628996Z * [new branch] gh/jansel/569/base -> origin/gh/jansel/569/base 2025-12-04T08:57:44.2630079Z * [new branch] gh/jansel/569/head -> origin/gh/jansel/569/head 2025-12-04T08:57:44.2631192Z * [new branch] gh/jansel/569/orig -> origin/gh/jansel/569/orig 2025-12-04T08:57:44.2632697Z * [new branch] gh/jansel/570/base -> origin/gh/jansel/570/base 2025-12-04T08:57:44.2633825Z * [new branch] gh/jansel/570/head -> origin/gh/jansel/570/head 2025-12-04T08:57:44.2634968Z * [new branch] gh/jansel/570/orig -> origin/gh/jansel/570/orig 2025-12-04T08:57:44.2636617Z * [new branch] gh/jansel/571/base -> origin/gh/jansel/571/base 2025-12-04T08:57:44.2637741Z * [new branch] gh/jansel/571/head -> origin/gh/jansel/571/head 2025-12-04T08:57:44.2638869Z * [new branch] gh/jansel/571/orig -> origin/gh/jansel/571/orig 2025-12-04T08:57:44.2640391Z * [new branch] gh/jansel/572/base -> origin/gh/jansel/572/base 2025-12-04T08:57:44.2641533Z * [new branch] gh/jansel/572/head -> origin/gh/jansel/572/head 2025-12-04T08:57:44.2642625Z * [new branch] gh/jansel/572/orig -> origin/gh/jansel/572/orig 2025-12-04T08:57:44.2644176Z * [new branch] gh/jansel/573/base -> origin/gh/jansel/573/base 2025-12-04T08:57:44.2645252Z * [new branch] gh/jansel/573/head -> origin/gh/jansel/573/head 2025-12-04T08:57:44.2646434Z * [new branch] gh/jansel/573/orig -> origin/gh/jansel/573/orig 2025-12-04T08:57:44.2647971Z * [new branch] gh/jansel/574/base -> origin/gh/jansel/574/base 2025-12-04T08:57:44.2649071Z * [new branch] gh/jansel/574/head -> origin/gh/jansel/574/head 2025-12-04T08:57:44.2650175Z * [new branch] gh/jansel/574/orig -> origin/gh/jansel/574/orig 2025-12-04T08:57:44.2651826Z * [new branch] gh/jansel/575/base -> origin/gh/jansel/575/base 2025-12-04T08:57:44.2652908Z * [new branch] gh/jansel/575/head -> origin/gh/jansel/575/head 2025-12-04T08:57:44.2653966Z * [new branch] gh/jansel/575/orig -> origin/gh/jansel/575/orig 2025-12-04T08:57:44.2655500Z * [new branch] gh/jansel/576/base -> origin/gh/jansel/576/base 2025-12-04T08:57:44.2656591Z * [new branch] gh/jansel/576/head -> origin/gh/jansel/576/head 2025-12-04T08:57:44.2657657Z * [new branch] gh/jansel/576/orig -> origin/gh/jansel/576/orig 2025-12-04T08:57:44.2659502Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-12-04T08:57:44.2661043Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-12-04T08:57:44.2662166Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-12-04T08:57:44.2663666Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-12-04T08:57:44.2664736Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-12-04T08:57:44.2665880Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-12-04T08:57:44.2668122Z * [new branch] gh/jerryzh168/1/base -> origin/gh/jerryzh168/1/base 2025-12-04T08:57:44.2669350Z * [new branch] gh/jerryzh168/1/head -> origin/gh/jerryzh168/1/head 2025-12-04T08:57:44.2670279Z * [new branch] gh/jerryzh168/1/orig -> origin/gh/jerryzh168/1/orig 2025-12-04T08:57:44.2672154Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-12-04T08:57:44.2673279Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-12-04T08:57:44.2674413Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-12-04T08:57:44.2675842Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-12-04T08:57:44.2676975Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-12-04T08:57:44.2678105Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-12-04T08:57:44.2679580Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-12-04T08:57:44.2680869Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-12-04T08:57:44.2681994Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-12-04T08:57:44.2683424Z * [new branch] gh/jiayisunx/77/base -> origin/gh/jiayisunx/77/base 2025-12-04T08:57:44.2684545Z * [new branch] gh/jiayisunx/77/head -> origin/gh/jiayisunx/77/head 2025-12-04T08:57:44.2685608Z * [new branch] gh/jiayisunx/77/orig -> origin/gh/jiayisunx/77/orig 2025-12-04T08:57:44.2687205Z * [new branch] gh/jiayisunx/78/base -> origin/gh/jiayisunx/78/base 2025-12-04T08:57:44.2688271Z * [new branch] gh/jiayisunx/78/head -> origin/gh/jiayisunx/78/head 2025-12-04T08:57:44.2689366Z * [new branch] gh/jiayisunx/78/orig -> origin/gh/jiayisunx/78/orig 2025-12-04T08:57:44.2690798Z * [new branch] gh/jiayisunx/79/base -> origin/gh/jiayisunx/79/base 2025-12-04T08:57:44.2691939Z * [new branch] gh/jiayisunx/79/head -> origin/gh/jiayisunx/79/head 2025-12-04T08:57:44.2693137Z * [new branch] gh/jiayisunx/79/orig -> origin/gh/jiayisunx/79/orig 2025-12-04T08:57:44.2694635Z * [new branch] gh/jiayisunx/82/base -> origin/gh/jiayisunx/82/base 2025-12-04T08:57:44.2695837Z * [new branch] gh/jiayisunx/82/head -> origin/gh/jiayisunx/82/head 2025-12-04T08:57:44.2696920Z * [new branch] gh/jiayisunx/82/orig -> origin/gh/jiayisunx/82/orig 2025-12-04T08:57:44.2698335Z * [new branch] gh/jiayisunx/83/base -> origin/gh/jiayisunx/83/base 2025-12-04T08:57:44.2699458Z * [new branch] gh/jiayisunx/83/head -> origin/gh/jiayisunx/83/head 2025-12-04T08:57:44.2700521Z * [new branch] gh/jiayisunx/83/orig -> origin/gh/jiayisunx/83/orig 2025-12-04T08:57:44.2701921Z * [new branch] gh/jiayisunx/84/base -> origin/gh/jiayisunx/84/base 2025-12-04T08:57:44.2703037Z * [new branch] gh/jiayisunx/84/head -> origin/gh/jiayisunx/84/head 2025-12-04T08:57:44.2704141Z * [new branch] gh/jiayisunx/84/orig -> origin/gh/jiayisunx/84/orig 2025-12-04T08:57:44.2705562Z * [new branch] gh/jiayisunx/85/base -> origin/gh/jiayisunx/85/base 2025-12-04T08:57:44.2706653Z * [new branch] gh/jiayisunx/85/head -> origin/gh/jiayisunx/85/head 2025-12-04T08:57:44.2708028Z * [new branch] gh/jiayisunx/85/orig -> origin/gh/jiayisunx/85/orig 2025-12-04T08:57:44.2709473Z * [new branch] gh/jiayisunx/86/base -> origin/gh/jiayisunx/86/base 2025-12-04T08:57:44.2710757Z * [new branch] gh/jiayisunx/86/head -> origin/gh/jiayisunx/86/head 2025-12-04T08:57:44.2711905Z * [new branch] gh/jiayisunx/86/orig -> origin/gh/jiayisunx/86/orig 2025-12-04T08:57:44.2713536Z * [new branch] gh/jiayisunx/87/base -> origin/gh/jiayisunx/87/base 2025-12-04T08:57:44.2714482Z * [new branch] gh/jiayisunx/87/head -> origin/gh/jiayisunx/87/head 2025-12-04T08:57:44.2715673Z * [new branch] gh/jiayisunx/87/orig -> origin/gh/jiayisunx/87/orig 2025-12-04T08:57:44.2717156Z * [new branch] gh/jiayisunx/88/base -> origin/gh/jiayisunx/88/base 2025-12-04T08:57:44.2718230Z * [new branch] gh/jiayisunx/88/head -> origin/gh/jiayisunx/88/head 2025-12-04T08:57:44.2719378Z * [new branch] gh/jiayisunx/88/orig -> origin/gh/jiayisunx/88/orig 2025-12-04T08:57:44.2720919Z * [new branch] gh/jiayisunx/89/base -> origin/gh/jiayisunx/89/base 2025-12-04T08:57:44.2722015Z * [new branch] gh/jiayisunx/89/head -> origin/gh/jiayisunx/89/head 2025-12-04T08:57:44.2723116Z * [new branch] gh/jiayisunx/89/orig -> origin/gh/jiayisunx/89/orig 2025-12-04T08:57:44.2724978Z * [new branch] gh/jiayisunx/90/base -> origin/gh/jiayisunx/90/base 2025-12-04T08:57:44.2726260Z * [new branch] gh/jiayisunx/90/head -> origin/gh/jiayisunx/90/head 2025-12-04T08:57:44.2727378Z * [new branch] gh/jiayisunx/90/orig -> origin/gh/jiayisunx/90/orig 2025-12-04T08:57:44.2729059Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-12-04T08:57:44.2730141Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-12-04T08:57:44.2731912Z * [new branch] gh/jturney/1/base -> origin/gh/jturney/1/base 2025-12-04T08:57:44.2733019Z * [new branch] gh/jturney/1/head -> origin/gh/jturney/1/head 2025-12-04T08:57:44.2734146Z * [new branch] gh/jturney/1/orig -> origin/gh/jturney/1/orig 2025-12-04T08:57:44.2735673Z * [new branch] gh/jturney/2/base -> origin/gh/jturney/2/base 2025-12-04T08:57:44.2736887Z * [new branch] gh/jturney/2/head -> origin/gh/jturney/2/head 2025-12-04T08:57:44.2737962Z * [new branch] gh/jturney/2/orig -> origin/gh/jturney/2/orig 2025-12-04T08:57:44.2739993Z * [new branch] gh/karthickai/10/base -> origin/gh/karthickai/10/base 2025-12-04T08:57:44.2741179Z * [new branch] gh/karthickai/10/head -> origin/gh/karthickai/10/head 2025-12-04T08:57:44.2742303Z * [new branch] gh/karthickai/10/orig -> origin/gh/karthickai/10/orig 2025-12-04T08:57:44.2743720Z * [new branch] gh/karthickai/11/base -> origin/gh/karthickai/11/base 2025-12-04T08:57:44.2744901Z * [new branch] gh/karthickai/11/head -> origin/gh/karthickai/11/head 2025-12-04T08:57:44.2745981Z * [new branch] gh/karthickai/11/orig -> origin/gh/karthickai/11/orig 2025-12-04T08:57:44.2748179Z * [new branch] gh/karthickai/12/base -> origin/gh/karthickai/12/base 2025-12-04T08:57:44.2749418Z * [new branch] gh/karthickai/12/head -> origin/gh/karthickai/12/head 2025-12-04T08:57:44.2750601Z * [new branch] gh/karthickai/12/orig -> origin/gh/karthickai/12/orig 2025-12-04T08:57:44.2752156Z * [new branch] gh/karthickai/13/base -> origin/gh/karthickai/13/base 2025-12-04T08:57:44.2753375Z * [new branch] gh/karthickai/13/head -> origin/gh/karthickai/13/head 2025-12-04T08:57:44.2754489Z * [new branch] gh/karthickai/13/orig -> origin/gh/karthickai/13/orig 2025-12-04T08:57:44.2756329Z * [new branch] gh/karthickai/14/base -> origin/gh/karthickai/14/base 2025-12-04T08:57:44.2757546Z * [new branch] gh/karthickai/14/head -> origin/gh/karthickai/14/head 2025-12-04T08:57:44.2758668Z * [new branch] gh/karthickai/14/orig -> origin/gh/karthickai/14/orig 2025-12-04T08:57:44.2760788Z * [new branch] gh/karthickai/15/base -> origin/gh/karthickai/15/base 2025-12-04T08:57:44.2761864Z * [new branch] gh/karthickai/15/head -> origin/gh/karthickai/15/head 2025-12-04T08:57:44.2762955Z * [new branch] gh/karthickai/15/orig -> origin/gh/karthickai/15/orig 2025-12-04T08:57:44.2764376Z * [new branch] gh/karthickai/16/base -> origin/gh/karthickai/16/base 2025-12-04T08:57:44.2765450Z * [new branch] gh/karthickai/16/head -> origin/gh/karthickai/16/head 2025-12-04T08:57:44.2766556Z * [new branch] gh/karthickai/16/orig -> origin/gh/karthickai/16/orig 2025-12-04T08:57:44.2768037Z * [new branch] gh/karthickai/17/base -> origin/gh/karthickai/17/base 2025-12-04T08:57:44.2769059Z * [new branch] gh/karthickai/17/head -> origin/gh/karthickai/17/head 2025-12-04T08:57:44.2770116Z * [new branch] gh/karthickai/17/orig -> origin/gh/karthickai/17/orig 2025-12-04T08:57:44.2772328Z * [new branch] gh/karthickai/18/base -> origin/gh/karthickai/18/base 2025-12-04T08:57:44.2773686Z * [new branch] gh/karthickai/18/head -> origin/gh/karthickai/18/head 2025-12-04T08:57:44.2775097Z * [new branch] gh/karthickai/18/orig -> origin/gh/karthickai/18/orig 2025-12-04T08:57:44.2776652Z * [new branch] gh/karthickai/19/base -> origin/gh/karthickai/19/base 2025-12-04T08:57:44.2777764Z * [new branch] gh/karthickai/19/head -> origin/gh/karthickai/19/head 2025-12-04T08:57:44.2778830Z * [new branch] gh/karthickai/19/orig -> origin/gh/karthickai/19/orig 2025-12-04T08:57:44.2781066Z * [new branch] gh/karthickai/20/base -> origin/gh/karthickai/20/base 2025-12-04T08:57:44.2782957Z * [new branch] gh/karthickai/20/head -> origin/gh/karthickai/20/head 2025-12-04T08:57:44.2784555Z * [new branch] gh/karthickai/20/orig -> origin/gh/karthickai/20/orig 2025-12-04T08:57:44.2786125Z * [new branch] gh/karthickai/21/base -> origin/gh/karthickai/21/base 2025-12-04T08:57:44.2787560Z * [new branch] gh/karthickai/21/head -> origin/gh/karthickai/21/head 2025-12-04T08:57:44.2789005Z * [new branch] gh/karthickai/21/orig -> origin/gh/karthickai/21/orig 2025-12-04T08:57:44.2790823Z * [new branch] gh/karthickai/22/base -> origin/gh/karthickai/22/base 2025-12-04T08:57:44.2791912Z * [new branch] gh/karthickai/22/head -> origin/gh/karthickai/22/head 2025-12-04T08:57:44.2793021Z * [new branch] gh/karthickai/22/orig -> origin/gh/karthickai/22/orig 2025-12-04T08:57:44.2794690Z * [new branch] gh/karthickai/23/base -> origin/gh/karthickai/23/base 2025-12-04T08:57:44.2796020Z * [new branch] gh/karthickai/23/head -> origin/gh/karthickai/23/head 2025-12-04T08:57:44.2797174Z * [new branch] gh/karthickai/23/orig -> origin/gh/karthickai/23/orig 2025-12-04T08:57:44.2798658Z * [new branch] gh/karthickai/24/base -> origin/gh/karthickai/24/base 2025-12-04T08:57:44.2799889Z * [new branch] gh/karthickai/24/head -> origin/gh/karthickai/24/head 2025-12-04T08:57:44.2801033Z * [new branch] gh/karthickai/24/orig -> origin/gh/karthickai/24/orig 2025-12-04T08:57:44.2803148Z * [new branch] gh/karthickai/25/base -> origin/gh/karthickai/25/base 2025-12-04T08:57:44.2804363Z * [new branch] gh/karthickai/25/head -> origin/gh/karthickai/25/head 2025-12-04T08:57:44.2805434Z * [new branch] gh/karthickai/25/orig -> origin/gh/karthickai/25/orig 2025-12-04T08:57:44.2806920Z * [new branch] gh/karthickai/26/base -> origin/gh/karthickai/26/base 2025-12-04T08:57:44.2808202Z * [new branch] gh/karthickai/26/head -> origin/gh/karthickai/26/head 2025-12-04T08:57:44.2809315Z * [new branch] gh/karthickai/26/orig -> origin/gh/karthickai/26/orig 2025-12-04T08:57:44.2812335Z * [new branch] gh/karthickai/6/base -> origin/gh/karthickai/6/base 2025-12-04T08:57:44.2814011Z * [new branch] gh/karthickai/6/head -> origin/gh/karthickai/6/head 2025-12-04T08:57:44.2815205Z * [new branch] gh/karthickai/6/orig -> origin/gh/karthickai/6/orig 2025-12-04T08:57:44.2816913Z * [new branch] gh/krocki/1/base -> origin/gh/krocki/1/base 2025-12-04T08:57:44.2818090Z * [new branch] gh/krocki/1/head -> origin/gh/krocki/1/head 2025-12-04T08:57:44.2819200Z * [new branch] gh/krocki/1/orig -> origin/gh/krocki/1/orig 2025-12-04T08:57:44.2820758Z * [new branch] gh/krocki/2/base -> origin/gh/krocki/2/base 2025-12-04T08:57:44.2821845Z * [new branch] gh/krocki/2/head -> origin/gh/krocki/2/head 2025-12-04T08:57:44.2823040Z * [new branch] gh/krocki/2/orig -> origin/gh/krocki/2/orig 2025-12-04T08:57:44.2827817Z * [new branch] gh/kurtamohler/60/base -> origin/gh/kurtamohler/60/base 2025-12-04T08:57:44.2829086Z * [new branch] gh/kurtamohler/60/head -> origin/gh/kurtamohler/60/head 2025-12-04T08:57:44.2830324Z * [new branch] gh/kurtamohler/60/orig -> origin/gh/kurtamohler/60/orig 2025-12-04T08:57:44.2831856Z * [new branch] gh/kurtamohler/61/base -> origin/gh/kurtamohler/61/base 2025-12-04T08:57:44.2832990Z * [new branch] gh/kurtamohler/61/head -> origin/gh/kurtamohler/61/head 2025-12-04T08:57:44.2834144Z * [new branch] gh/kurtamohler/61/orig -> origin/gh/kurtamohler/61/orig 2025-12-04T08:57:44.2835776Z * [new branch] gh/kurtamohler/62/base -> origin/gh/kurtamohler/62/base 2025-12-04T08:57:44.2836914Z * [new branch] gh/kurtamohler/62/head -> origin/gh/kurtamohler/62/head 2025-12-04T08:57:44.2838006Z * [new branch] gh/kurtamohler/62/orig -> origin/gh/kurtamohler/62/orig 2025-12-04T08:57:44.2839619Z * [new branch] gh/kurtamohler/63/base -> origin/gh/kurtamohler/63/base 2025-12-04T08:57:44.2840838Z * [new branch] gh/kurtamohler/63/head -> origin/gh/kurtamohler/63/head 2025-12-04T08:57:44.2841908Z * [new branch] gh/kurtamohler/63/orig -> origin/gh/kurtamohler/63/orig 2025-12-04T08:57:44.2843434Z * [new branch] gh/kurtamohler/64/base -> origin/gh/kurtamohler/64/base 2025-12-04T08:57:44.2844534Z * [new branch] gh/kurtamohler/64/head -> origin/gh/kurtamohler/64/head 2025-12-04T08:57:44.2846199Z * [new branch] gh/kurtamohler/64/orig -> origin/gh/kurtamohler/64/orig 2025-12-04T08:57:44.2847679Z * [new branch] gh/kurtamohler/65/base -> origin/gh/kurtamohler/65/base 2025-12-04T08:57:44.2848801Z * [new branch] gh/kurtamohler/65/head -> origin/gh/kurtamohler/65/head 2025-12-04T08:57:44.2849905Z * [new branch] gh/kurtamohler/65/orig -> origin/gh/kurtamohler/65/orig 2025-12-04T08:57:44.2851291Z * [new branch] gh/kurtamohler/66/base -> origin/gh/kurtamohler/66/base 2025-12-04T08:57:44.2852918Z * [new branch] gh/kurtamohler/66/head -> origin/gh/kurtamohler/66/head 2025-12-04T08:57:44.2853947Z * [new branch] gh/kurtamohler/66/orig -> origin/gh/kurtamohler/66/orig 2025-12-04T08:57:44.2855414Z * [new branch] gh/kurtamohler/67/base -> origin/gh/kurtamohler/67/base 2025-12-04T08:57:44.2856580Z * [new branch] gh/kurtamohler/67/head -> origin/gh/kurtamohler/67/head 2025-12-04T08:57:44.2857670Z * [new branch] gh/kurtamohler/67/orig -> origin/gh/kurtamohler/67/orig 2025-12-04T08:57:44.2859540Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-12-04T08:57:44.2860912Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-12-04T08:57:44.2861935Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-12-04T08:57:44.2863468Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-12-04T08:57:44.2864554Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-12-04T08:57:44.2866157Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-12-04T08:57:44.2867326Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-12-04T08:57:44.2868816Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-12-04T08:57:44.2870285Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-12-04T08:57:44.2871577Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-12-04T08:57:44.2872683Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-12-04T08:57:44.2874177Z * [new branch] gh/kwen2501/211/base -> origin/gh/kwen2501/211/base 2025-12-04T08:57:44.2875291Z * [new branch] gh/kwen2501/211/head -> origin/gh/kwen2501/211/head 2025-12-04T08:57:44.2876757Z * [new branch] gh/kwen2501/224/base -> origin/gh/kwen2501/224/base 2025-12-04T08:57:44.2877886Z * [new branch] gh/kwen2501/224/head -> origin/gh/kwen2501/224/head 2025-12-04T08:57:44.2879006Z * [new branch] gh/kwen2501/224/orig -> origin/gh/kwen2501/224/orig 2025-12-04T08:57:44.2880627Z * [new branch] gh/kwen2501/228/base -> origin/gh/kwen2501/228/base 2025-12-04T08:57:44.2881693Z * [new branch] gh/kwen2501/228/head -> origin/gh/kwen2501/228/head 2025-12-04T08:57:44.2882783Z * [new branch] gh/kwen2501/228/orig -> origin/gh/kwen2501/228/orig 2025-12-04T08:57:44.2884368Z * [new branch] gh/kwen2501/234/base -> origin/gh/kwen2501/234/base 2025-12-04T08:57:44.2885478Z * [new branch] gh/kwen2501/234/head -> origin/gh/kwen2501/234/head 2025-12-04T08:57:44.2886655Z * [new branch] gh/kwen2501/234/orig -> origin/gh/kwen2501/234/orig 2025-12-04T08:57:44.2888100Z * [new branch] gh/kwen2501/235/base -> origin/gh/kwen2501/235/base 2025-12-04T08:57:44.2889183Z * [new branch] gh/kwen2501/235/head -> origin/gh/kwen2501/235/head 2025-12-04T08:57:44.2890328Z * [new branch] gh/kwen2501/235/orig -> origin/gh/kwen2501/235/orig 2025-12-04T08:57:44.2891737Z * [new branch] gh/kwen2501/236/base -> origin/gh/kwen2501/236/base 2025-12-04T08:57:44.2892816Z * [new branch] gh/kwen2501/236/head -> origin/gh/kwen2501/236/head 2025-12-04T08:57:44.2893936Z * [new branch] gh/kwen2501/236/orig -> origin/gh/kwen2501/236/orig 2025-12-04T08:57:44.2895350Z * [new branch] gh/kwen2501/237/base -> origin/gh/kwen2501/237/base 2025-12-04T08:57:44.2896414Z * [new branch] gh/kwen2501/237/head -> origin/gh/kwen2501/237/head 2025-12-04T08:57:44.2897520Z * [new branch] gh/kwen2501/237/orig -> origin/gh/kwen2501/237/orig 2025-12-04T08:57:44.2898965Z * [new branch] gh/kwen2501/238/base -> origin/gh/kwen2501/238/base 2025-12-04T08:57:44.2900077Z * [new branch] gh/kwen2501/238/head -> origin/gh/kwen2501/238/head 2025-12-04T08:57:44.2901417Z * [new branch] gh/kwen2501/238/orig -> origin/gh/kwen2501/238/orig 2025-12-04T08:57:44.2902909Z * [new branch] gh/kwen2501/240/base -> origin/gh/kwen2501/240/base 2025-12-04T08:57:44.2903998Z * [new branch] gh/kwen2501/240/head -> origin/gh/kwen2501/240/head 2025-12-04T08:57:44.2905572Z * [new branch] gh/kwen2501/240/orig -> origin/gh/kwen2501/240/orig 2025-12-04T08:57:44.2906976Z * [new branch] gh/kwen2501/241/base -> origin/gh/kwen2501/241/base 2025-12-04T08:57:44.2908389Z * [new branch] gh/kwen2501/241/head -> origin/gh/kwen2501/241/head 2025-12-04T08:57:44.2909534Z * [new branch] gh/kwen2501/241/orig -> origin/gh/kwen2501/241/orig 2025-12-04T08:57:44.2911059Z * [new branch] gh/kwen2501/247/base -> origin/gh/kwen2501/247/base 2025-12-04T08:57:44.2912221Z * [new branch] gh/kwen2501/247/head -> origin/gh/kwen2501/247/head 2025-12-04T08:57:44.2913329Z * [new branch] gh/kwen2501/247/orig -> origin/gh/kwen2501/247/orig 2025-12-04T08:57:44.2915026Z * [new branch] gh/kwen2501/252/base -> origin/gh/kwen2501/252/base 2025-12-04T08:57:44.2915871Z * [new branch] gh/kwen2501/252/head -> origin/gh/kwen2501/252/head 2025-12-04T08:57:44.2917149Z * [new branch] gh/kwen2501/252/orig -> origin/gh/kwen2501/252/orig 2025-12-04T08:57:44.2919075Z * [new branch] gh/kwen2501/259/base -> origin/gh/kwen2501/259/base 2025-12-04T08:57:44.2920395Z * [new branch] gh/kwen2501/259/head -> origin/gh/kwen2501/259/head 2025-12-04T08:57:44.2921562Z * [new branch] gh/kwen2501/259/orig -> origin/gh/kwen2501/259/orig 2025-12-04T08:57:44.2923111Z * [new branch] gh/kwen2501/260/base -> origin/gh/kwen2501/260/base 2025-12-04T08:57:44.2924746Z * [new branch] gh/kwen2501/260/head -> origin/gh/kwen2501/260/head 2025-12-04T08:57:44.2925889Z * [new branch] gh/kwen2501/260/orig -> origin/gh/kwen2501/260/orig 2025-12-04T08:57:44.2927931Z * [new branch] gh/kwen2501/268/base -> origin/gh/kwen2501/268/base 2025-12-04T08:57:44.2929080Z * [new branch] gh/kwen2501/268/head -> origin/gh/kwen2501/268/head 2025-12-04T08:57:44.2930214Z * [new branch] gh/kwen2501/268/orig -> origin/gh/kwen2501/268/orig 2025-12-04T08:57:44.2931768Z * [new branch] gh/kwen2501/269/base -> origin/gh/kwen2501/269/base 2025-12-04T08:57:44.2932992Z * [new branch] gh/kwen2501/269/head -> origin/gh/kwen2501/269/head 2025-12-04T08:57:44.2934226Z * [new branch] gh/kwen2501/269/orig -> origin/gh/kwen2501/269/orig 2025-12-04T08:57:44.2935866Z * [new branch] gh/kwen2501/270/base -> origin/gh/kwen2501/270/base 2025-12-04T08:57:44.2937194Z * [new branch] gh/kwen2501/270/head -> origin/gh/kwen2501/270/head 2025-12-04T08:57:44.2938304Z * [new branch] gh/kwen2501/270/orig -> origin/gh/kwen2501/270/orig 2025-12-04T08:57:44.2939823Z * [new branch] gh/kwen2501/271/base -> origin/gh/kwen2501/271/base 2025-12-04T08:57:44.2940987Z * [new branch] gh/kwen2501/271/head -> origin/gh/kwen2501/271/head 2025-12-04T08:57:44.2942092Z * [new branch] gh/kwen2501/271/orig -> origin/gh/kwen2501/271/orig 2025-12-04T08:57:44.2943703Z * [new branch] gh/kwen2501/274/base -> origin/gh/kwen2501/274/base 2025-12-04T08:57:44.2944924Z * [new branch] gh/kwen2501/274/head -> origin/gh/kwen2501/274/head 2025-12-04T08:57:44.2946064Z * [new branch] gh/kwen2501/274/orig -> origin/gh/kwen2501/274/orig 2025-12-04T08:57:44.2948008Z * [new branch] gh/kwen2501/275/base -> origin/gh/kwen2501/275/base 2025-12-04T08:57:44.2949350Z * [new branch] gh/kwen2501/275/head -> origin/gh/kwen2501/275/head 2025-12-04T08:57:44.2950646Z * [new branch] gh/kwen2501/275/orig -> origin/gh/kwen2501/275/orig 2025-12-04T08:57:44.2952148Z * [new branch] gh/kwen2501/276/base -> origin/gh/kwen2501/276/base 2025-12-04T08:57:44.2953898Z * [new branch] gh/kwen2501/276/head -> origin/gh/kwen2501/276/head 2025-12-04T08:57:44.2954935Z * [new branch] gh/kwen2501/276/orig -> origin/gh/kwen2501/276/orig 2025-12-04T08:57:44.2956430Z * [new branch] gh/kwen2501/277/base -> origin/gh/kwen2501/277/base 2025-12-04T08:57:44.2957586Z * [new branch] gh/kwen2501/277/head -> origin/gh/kwen2501/277/head 2025-12-04T08:57:44.2958718Z * [new branch] gh/kwen2501/277/orig -> origin/gh/kwen2501/277/orig 2025-12-04T08:57:44.2960360Z * [new branch] gh/kwen2501/278/base -> origin/gh/kwen2501/278/base 2025-12-04T08:57:44.2961444Z * [new branch] gh/kwen2501/278/head -> origin/gh/kwen2501/278/head 2025-12-04T08:57:44.2962551Z * [new branch] gh/kwen2501/278/orig -> origin/gh/kwen2501/278/orig 2025-12-04T08:57:44.2964114Z * [new branch] gh/kwen2501/279/base -> origin/gh/kwen2501/279/base 2025-12-04T08:57:44.2965359Z * [new branch] gh/kwen2501/279/head -> origin/gh/kwen2501/279/head 2025-12-04T08:57:44.2966624Z * [new branch] gh/kwen2501/279/orig -> origin/gh/kwen2501/279/orig 2025-12-04T08:57:44.2968159Z * [new branch] gh/kwen2501/280/base -> origin/gh/kwen2501/280/base 2025-12-04T08:57:44.2969464Z * [new branch] gh/kwen2501/280/head -> origin/gh/kwen2501/280/head 2025-12-04T08:57:44.2970649Z * [new branch] gh/kwen2501/280/orig -> origin/gh/kwen2501/280/orig 2025-12-04T08:57:44.2972142Z * [new branch] gh/kwen2501/281/base -> origin/gh/kwen2501/281/base 2025-12-04T08:57:44.2973220Z * [new branch] gh/kwen2501/281/head -> origin/gh/kwen2501/281/head 2025-12-04T08:57:44.2974291Z * [new branch] gh/kwen2501/281/orig -> origin/gh/kwen2501/281/orig 2025-12-04T08:57:44.2975799Z * [new branch] gh/kwen2501/282/base -> origin/gh/kwen2501/282/base 2025-12-04T08:57:44.2976985Z * [new branch] gh/kwen2501/282/head -> origin/gh/kwen2501/282/head 2025-12-04T08:57:44.2978147Z * [new branch] gh/kwen2501/282/orig -> origin/gh/kwen2501/282/orig 2025-12-04T08:57:44.2979607Z * [new branch] gh/kwen2501/283/base -> origin/gh/kwen2501/283/base 2025-12-04T08:57:44.2980739Z * [new branch] gh/kwen2501/283/head -> origin/gh/kwen2501/283/head 2025-12-04T08:57:44.2981961Z * [new branch] gh/kwen2501/283/orig -> origin/gh/kwen2501/283/orig 2025-12-04T08:57:44.2983490Z * [new branch] gh/kwen2501/284/base -> origin/gh/kwen2501/284/base 2025-12-04T08:57:44.2984756Z * [new branch] gh/kwen2501/284/head -> origin/gh/kwen2501/284/head 2025-12-04T08:57:44.2985827Z * [new branch] gh/kwen2501/284/orig -> origin/gh/kwen2501/284/orig 2025-12-04T08:57:44.2987391Z * [new branch] gh/kwen2501/285/base -> origin/gh/kwen2501/285/base 2025-12-04T08:57:44.2988821Z * [new branch] gh/kwen2501/285/head -> origin/gh/kwen2501/285/head 2025-12-04T08:57:44.2989949Z * [new branch] gh/kwen2501/285/orig -> origin/gh/kwen2501/285/orig 2025-12-04T08:57:44.2991542Z * [new branch] gh/kwen2501/286/base -> origin/gh/kwen2501/286/base 2025-12-04T08:57:44.2992723Z * [new branch] gh/kwen2501/286/head -> origin/gh/kwen2501/286/head 2025-12-04T08:57:44.2993947Z * [new branch] gh/kwen2501/286/orig -> origin/gh/kwen2501/286/orig 2025-12-04T08:57:44.2995315Z * [new branch] gh/kwen2501/287/base -> origin/gh/kwen2501/287/base 2025-12-04T08:57:44.2996438Z * [new branch] gh/kwen2501/287/head -> origin/gh/kwen2501/287/head 2025-12-04T08:57:44.2997657Z * [new branch] gh/kwen2501/287/orig -> origin/gh/kwen2501/287/orig 2025-12-04T08:57:44.2999321Z * [new branch] gh/kwen2501/288/base -> origin/gh/kwen2501/288/base 2025-12-04T08:57:44.3000370Z * [new branch] gh/kwen2501/288/head -> origin/gh/kwen2501/288/head 2025-12-04T08:57:44.3001576Z * [new branch] gh/kwen2501/288/orig -> origin/gh/kwen2501/288/orig 2025-12-04T08:57:44.3003311Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-12-04T08:57:44.3004389Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-12-04T08:57:44.3005450Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-12-04T08:57:44.3006880Z * [new branch] gh/laithsakka/276/base -> origin/gh/laithsakka/276/base 2025-12-04T08:57:44.3007960Z * [new branch] gh/laithsakka/276/head -> origin/gh/laithsakka/276/head 2025-12-04T08:57:44.3009083Z * [new branch] gh/laithsakka/276/orig -> origin/gh/laithsakka/276/orig 2025-12-04T08:57:44.3010684Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-12-04T08:57:44.3012120Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-12-04T08:57:44.3013387Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-12-04T08:57:44.3014499Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-12-04T08:57:44.3015776Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-12-04T08:57:44.3016779Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-12-04T08:57:44.3018359Z * [new branch] gh/laithsakka/313/base -> origin/gh/laithsakka/313/base 2025-12-04T08:57:44.3019424Z * [new branch] gh/laithsakka/313/head -> origin/gh/laithsakka/313/head 2025-12-04T08:57:44.3020568Z * [new branch] gh/laithsakka/313/orig -> origin/gh/laithsakka/313/orig 2025-12-04T08:57:44.3022332Z * [new branch] gh/laithsakka/316/base -> origin/gh/laithsakka/316/base 2025-12-04T08:57:44.3023394Z * [new branch] gh/laithsakka/316/head -> origin/gh/laithsakka/316/head 2025-12-04T08:57:44.3024870Z * [new branch] gh/laithsakka/316/orig -> origin/gh/laithsakka/316/orig 2025-12-04T08:57:44.3026524Z * [new branch] gh/laithsakka/317/base -> origin/gh/laithsakka/317/base 2025-12-04T08:57:44.3027695Z * [new branch] gh/laithsakka/317/head -> origin/gh/laithsakka/317/head 2025-12-04T08:57:44.3028853Z * [new branch] gh/laithsakka/317/orig -> origin/gh/laithsakka/317/orig 2025-12-04T08:57:44.3030479Z * [new branch] gh/laithsakka/319/base -> origin/gh/laithsakka/319/base 2025-12-04T08:57:44.3031644Z * [new branch] gh/laithsakka/319/head -> origin/gh/laithsakka/319/head 2025-12-04T08:57:44.3032760Z * [new branch] gh/laithsakka/319/orig -> origin/gh/laithsakka/319/orig 2025-12-04T08:57:44.3034130Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-12-04T08:57:44.3035741Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-12-04T08:57:44.3037363Z * [new branch] gh/laithsakka/320/base -> origin/gh/laithsakka/320/base 2025-12-04T08:57:44.3038401Z * [new branch] gh/laithsakka/320/head -> origin/gh/laithsakka/320/head 2025-12-04T08:57:44.3039634Z * [new branch] gh/laithsakka/320/orig -> origin/gh/laithsakka/320/orig 2025-12-04T08:57:44.3041195Z * [new branch] gh/laithsakka/321/base -> origin/gh/laithsakka/321/base 2025-12-04T08:57:44.3042263Z * [new branch] gh/laithsakka/321/head -> origin/gh/laithsakka/321/head 2025-12-04T08:57:44.3043475Z * [new branch] gh/laithsakka/321/orig -> origin/gh/laithsakka/321/orig 2025-12-04T08:57:44.3045067Z * [new branch] gh/laithsakka/322/base -> origin/gh/laithsakka/322/base 2025-12-04T08:57:44.3046160Z * [new branch] gh/laithsakka/322/head -> origin/gh/laithsakka/322/head 2025-12-04T08:57:44.3047218Z * [new branch] gh/laithsakka/322/orig -> origin/gh/laithsakka/322/orig 2025-12-04T08:57:44.3048780Z * [new branch] gh/laithsakka/323/base -> origin/gh/laithsakka/323/base 2025-12-04T08:57:44.3050008Z * [new branch] gh/laithsakka/323/head -> origin/gh/laithsakka/323/head 2025-12-04T08:57:44.3051119Z * [new branch] gh/laithsakka/323/orig -> origin/gh/laithsakka/323/orig 2025-12-04T08:57:44.3053127Z * [new branch] gh/laithsakka/324/base -> origin/gh/laithsakka/324/base 2025-12-04T08:57:44.3054158Z * [new branch] gh/laithsakka/324/head -> origin/gh/laithsakka/324/head 2025-12-04T08:57:44.3055250Z * [new branch] gh/laithsakka/324/orig -> origin/gh/laithsakka/324/orig 2025-12-04T08:57:44.3056979Z * [new branch] gh/laithsakka/325/base -> origin/gh/laithsakka/325/base 2025-12-04T08:57:44.3058105Z * [new branch] gh/laithsakka/325/head -> origin/gh/laithsakka/325/head 2025-12-04T08:57:44.3059236Z * [new branch] gh/laithsakka/325/orig -> origin/gh/laithsakka/325/orig 2025-12-04T08:57:44.3060957Z * [new branch] gh/laithsakka/326/base -> origin/gh/laithsakka/326/base 2025-12-04T08:57:44.3062041Z * [new branch] gh/laithsakka/326/head -> origin/gh/laithsakka/326/head 2025-12-04T08:57:44.3063144Z * [new branch] gh/laithsakka/326/orig -> origin/gh/laithsakka/326/orig 2025-12-04T08:57:44.3064701Z * [new branch] gh/laithsakka/327/base -> origin/gh/laithsakka/327/base 2025-12-04T08:57:44.3065906Z * [new branch] gh/laithsakka/327/head -> origin/gh/laithsakka/327/head 2025-12-04T08:57:44.3067053Z * [new branch] gh/laithsakka/327/orig -> origin/gh/laithsakka/327/orig 2025-12-04T08:57:44.3068879Z * [new branch] gh/laithsakka/328/base -> origin/gh/laithsakka/328/base 2025-12-04T08:57:44.3070007Z * [new branch] gh/laithsakka/328/head -> origin/gh/laithsakka/328/head 2025-12-04T08:57:44.3071585Z * [new branch] gh/laithsakka/328/orig -> origin/gh/laithsakka/328/orig 2025-12-04T08:57:44.3073853Z * [new branch] gh/liangel/4/base -> origin/gh/liangel/4/base 2025-12-04T08:57:44.3075014Z * [new branch] gh/liangel/4/head -> origin/gh/liangel/4/head 2025-12-04T08:57:44.3076154Z * [new branch] gh/liangel/4/orig -> origin/gh/liangel/4/orig 2025-12-04T08:57:44.3080115Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-12-04T08:57:44.3081230Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-12-04T08:57:44.3082922Z * [new branch] gh/lw/4/base -> origin/gh/lw/4/base 2025-12-04T08:57:44.3084034Z * [new branch] gh/lw/4/head -> origin/gh/lw/4/head 2025-12-04T08:57:44.3085171Z * [new branch] gh/lw/4/orig -> origin/gh/lw/4/orig 2025-12-04T08:57:44.3086650Z * [new branch] gh/lw/5/base -> origin/gh/lw/5/base 2025-12-04T08:57:44.3087677Z * [new branch] gh/lw/5/head -> origin/gh/lw/5/head 2025-12-04T08:57:44.3088877Z * [new branch] gh/lw/5/orig -> origin/gh/lw/5/orig 2025-12-04T08:57:44.3090290Z * [new branch] gh/lw/6/base -> origin/gh/lw/6/base 2025-12-04T08:57:44.3091359Z * [new branch] gh/lw/6/head -> origin/gh/lw/6/head 2025-12-04T08:57:44.3092457Z * [new branch] gh/lw/6/orig -> origin/gh/lw/6/orig 2025-12-04T08:57:44.3094261Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-12-04T08:57:44.3095635Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-12-04T08:57:44.3096702Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-12-04T08:57:44.3097859Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-12-04T08:57:44.3099316Z * [new branch] gh/malfet/506/base -> origin/gh/malfet/506/base 2025-12-04T08:57:44.3100387Z * [new branch] gh/malfet/506/head -> origin/gh/malfet/506/head 2025-12-04T08:57:44.3101484Z * [new branch] gh/malfet/506/orig -> origin/gh/malfet/506/orig 2025-12-04T08:57:44.3103066Z * [new branch] gh/malfet/517/base -> origin/gh/malfet/517/base 2025-12-04T08:57:44.3104169Z * [new branch] gh/malfet/517/head -> origin/gh/malfet/517/head 2025-12-04T08:57:44.3105719Z * [new branch] gh/malfet/528/base -> origin/gh/malfet/528/base 2025-12-04T08:57:44.3107427Z * [new branch] gh/malfet/528/head -> origin/gh/malfet/528/head 2025-12-04T08:57:44.3108807Z * [new branch] gh/malfet/528/orig -> origin/gh/malfet/528/orig 2025-12-04T08:57:44.3110350Z * [new branch] gh/malfet/537/base -> origin/gh/malfet/537/base 2025-12-04T08:57:44.3111539Z * [new branch] gh/malfet/537/head -> origin/gh/malfet/537/head 2025-12-04T08:57:44.3112659Z * [new branch] gh/malfet/537/orig -> origin/gh/malfet/537/orig 2025-12-04T08:57:44.3114166Z * [new branch] gh/malfet/546/base -> origin/gh/malfet/546/base 2025-12-04T08:57:44.3115283Z * [new branch] gh/malfet/546/head -> origin/gh/malfet/546/head 2025-12-04T08:57:44.3116399Z * [new branch] gh/malfet/546/orig -> origin/gh/malfet/546/orig 2025-12-04T08:57:44.3118050Z * [new branch] gh/malfet/565/base -> origin/gh/malfet/565/base 2025-12-04T08:57:44.3119090Z * [new branch] gh/malfet/565/head -> origin/gh/malfet/565/head 2025-12-04T08:57:44.3120336Z * [new branch] gh/malfet/565/orig -> origin/gh/malfet/565/orig 2025-12-04T08:57:44.3121815Z * [new branch] gh/malfet/575/base -> origin/gh/malfet/575/base 2025-12-04T08:57:44.3122886Z * [new branch] gh/malfet/575/head -> origin/gh/malfet/575/head 2025-12-04T08:57:44.3124330Z * [new branch] gh/malfet/575/orig -> origin/gh/malfet/575/orig 2025-12-04T08:57:44.3125943Z * [new branch] gh/malfet/580/base -> origin/gh/malfet/580/base 2025-12-04T08:57:44.3127044Z * [new branch] gh/malfet/580/head -> origin/gh/malfet/580/head 2025-12-04T08:57:44.3128140Z * [new branch] gh/malfet/580/orig -> origin/gh/malfet/580/orig 2025-12-04T08:57:44.3129650Z * [new branch] gh/malfet/581/base -> origin/gh/malfet/581/base 2025-12-04T08:57:44.3130767Z * [new branch] gh/malfet/581/head -> origin/gh/malfet/581/head 2025-12-04T08:57:44.3131874Z * [new branch] gh/malfet/581/orig -> origin/gh/malfet/581/orig 2025-12-04T08:57:44.3133367Z * [new branch] gh/malfet/583/base -> origin/gh/malfet/583/base 2025-12-04T08:57:44.3134488Z * [new branch] gh/malfet/583/head -> origin/gh/malfet/583/head 2025-12-04T08:57:44.3135691Z * [new branch] gh/malfet/583/orig -> origin/gh/malfet/583/orig 2025-12-04T08:57:44.3137211Z * [new branch] gh/malfet/586/base -> origin/gh/malfet/586/base 2025-12-04T08:57:44.3138373Z * [new branch] gh/malfet/586/head -> origin/gh/malfet/586/head 2025-12-04T08:57:44.3139544Z * [new branch] gh/malfet/586/orig -> origin/gh/malfet/586/orig 2025-12-04T08:57:44.3140955Z * [new branch] gh/malfet/587/base -> origin/gh/malfet/587/base 2025-12-04T08:57:44.3141985Z * [new branch] gh/malfet/587/head -> origin/gh/malfet/587/head 2025-12-04T08:57:44.3143055Z * [new branch] gh/malfet/587/orig -> origin/gh/malfet/587/orig 2025-12-04T08:57:44.3144908Z * [new branch] gh/malfet/588/base -> origin/gh/malfet/588/base 2025-12-04T08:57:44.3146058Z * [new branch] gh/malfet/588/head -> origin/gh/malfet/588/head 2025-12-04T08:57:44.3147275Z * [new branch] gh/malfet/588/orig -> origin/gh/malfet/588/orig 2025-12-04T08:57:44.3149172Z * [new branch] gh/malfet/589/base -> origin/gh/malfet/589/base 2025-12-04T08:57:44.3150269Z * [new branch] gh/malfet/589/head -> origin/gh/malfet/589/head 2025-12-04T08:57:44.3151398Z * [new branch] gh/malfet/589/orig -> origin/gh/malfet/589/orig 2025-12-04T08:57:44.3152851Z * [new branch] gh/malfet/590/base -> origin/gh/malfet/590/base 2025-12-04T08:57:44.3153996Z * [new branch] gh/malfet/590/head -> origin/gh/malfet/590/head 2025-12-04T08:57:44.3155108Z * [new branch] gh/malfet/590/orig -> origin/gh/malfet/590/orig 2025-12-04T08:57:44.3157065Z * [new branch] gh/malfet/591/base -> origin/gh/malfet/591/base 2025-12-04T08:57:44.3158172Z * [new branch] gh/malfet/591/head -> origin/gh/malfet/591/head 2025-12-04T08:57:44.3159377Z * [new branch] gh/malfet/591/orig -> origin/gh/malfet/591/orig 2025-12-04T08:57:44.3160971Z * [new branch] gh/malfet/592/base -> origin/gh/malfet/592/base 2025-12-04T08:57:44.3162054Z * [new branch] gh/malfet/592/head -> origin/gh/malfet/592/head 2025-12-04T08:57:44.3163137Z * [new branch] gh/malfet/592/orig -> origin/gh/malfet/592/orig 2025-12-04T08:57:44.3164676Z * [new branch] gh/malfet/593/base -> origin/gh/malfet/593/base 2025-12-04T08:57:44.3165767Z * [new branch] gh/malfet/593/head -> origin/gh/malfet/593/head 2025-12-04T08:57:44.3166859Z * [new branch] gh/malfet/593/orig -> origin/gh/malfet/593/orig 2025-12-04T08:57:44.3168378Z * [new branch] gh/malfet/594/base -> origin/gh/malfet/594/base 2025-12-04T08:57:44.3169502Z * [new branch] gh/malfet/594/head -> origin/gh/malfet/594/head 2025-12-04T08:57:44.3170611Z * [new branch] gh/malfet/594/orig -> origin/gh/malfet/594/orig 2025-12-04T08:57:44.3172166Z * [new branch] gh/malfet/595/base -> origin/gh/malfet/595/base 2025-12-04T08:57:44.3173324Z * [new branch] gh/malfet/595/head -> origin/gh/malfet/595/head 2025-12-04T08:57:44.3175226Z * [new branch] gh/malfet/595/orig -> origin/gh/malfet/595/orig 2025-12-04T08:57:44.3177145Z * [new branch] gh/malfet/596/base -> origin/gh/malfet/596/base 2025-12-04T08:57:44.3178236Z * [new branch] gh/malfet/596/head -> origin/gh/malfet/596/head 2025-12-04T08:57:44.3179335Z * [new branch] gh/malfet/596/orig -> origin/gh/malfet/596/orig 2025-12-04T08:57:44.3180932Z * [new branch] gh/malfet/597/base -> origin/gh/malfet/597/base 2025-12-04T08:57:44.3182043Z * [new branch] gh/malfet/597/head -> origin/gh/malfet/597/head 2025-12-04T08:57:44.3183124Z * [new branch] gh/malfet/597/orig -> origin/gh/malfet/597/orig 2025-12-04T08:57:44.3184655Z * [new branch] gh/malfet/598/base -> origin/gh/malfet/598/base 2025-12-04T08:57:44.3185732Z * [new branch] gh/malfet/598/head -> origin/gh/malfet/598/head 2025-12-04T08:57:44.3186964Z * [new branch] gh/malfet/598/orig -> origin/gh/malfet/598/orig 2025-12-04T08:57:44.3188747Z * [new branch] gh/malfet/599/base -> origin/gh/malfet/599/base 2025-12-04T08:57:44.3189818Z * [new branch] gh/malfet/599/head -> origin/gh/malfet/599/head 2025-12-04T08:57:44.3190951Z * [new branch] gh/malfet/599/orig -> origin/gh/malfet/599/orig 2025-12-04T08:57:44.3192430Z * [new branch] gh/malfet/600/base -> origin/gh/malfet/600/base 2025-12-04T08:57:44.3193526Z * [new branch] gh/malfet/600/head -> origin/gh/malfet/600/head 2025-12-04T08:57:44.3194648Z * [new branch] gh/malfet/600/orig -> origin/gh/malfet/600/orig 2025-12-04T08:57:44.3196259Z * [new branch] gh/malfet/601/base -> origin/gh/malfet/601/base 2025-12-04T08:57:44.3197497Z * [new branch] gh/malfet/601/head -> origin/gh/malfet/601/head 2025-12-04T08:57:44.3198652Z * [new branch] gh/malfet/601/orig -> origin/gh/malfet/601/orig 2025-12-04T08:57:44.3200316Z * [new branch] gh/malfet/602/base -> origin/gh/malfet/602/base 2025-12-04T08:57:44.3201408Z * [new branch] gh/malfet/602/head -> origin/gh/malfet/602/head 2025-12-04T08:57:44.3202509Z * [new branch] gh/malfet/602/orig -> origin/gh/malfet/602/orig 2025-12-04T08:57:44.3203862Z * [new branch] gh/malfet/603/base -> origin/gh/malfet/603/base 2025-12-04T08:57:44.3204954Z * [new branch] gh/malfet/603/head -> origin/gh/malfet/603/head 2025-12-04T08:57:44.3205970Z * [new branch] gh/malfet/603/orig -> origin/gh/malfet/603/orig 2025-12-04T08:57:44.3207437Z * [new branch] gh/malfet/604/base -> origin/gh/malfet/604/base 2025-12-04T08:57:44.3208512Z * [new branch] gh/malfet/604/head -> origin/gh/malfet/604/head 2025-12-04T08:57:44.3209598Z * [new branch] gh/malfet/604/orig -> origin/gh/malfet/604/orig 2025-12-04T08:57:44.3211286Z * [new branch] gh/malfet/605/base -> origin/gh/malfet/605/base 2025-12-04T08:57:44.3212417Z * [new branch] gh/malfet/605/head -> origin/gh/malfet/605/head 2025-12-04T08:57:44.3213587Z * [new branch] gh/malfet/605/orig -> origin/gh/malfet/605/orig 2025-12-04T08:57:44.3215092Z * [new branch] gh/malfet/606/base -> origin/gh/malfet/606/base 2025-12-04T08:57:44.3216250Z * [new branch] gh/malfet/606/head -> origin/gh/malfet/606/head 2025-12-04T08:57:44.3217423Z * [new branch] gh/malfet/606/orig -> origin/gh/malfet/606/orig 2025-12-04T08:57:44.3218892Z * [new branch] gh/malfet/607/base -> origin/gh/malfet/607/base 2025-12-04T08:57:44.3219955Z * [new branch] gh/malfet/607/head -> origin/gh/malfet/607/head 2025-12-04T08:57:44.3221132Z * [new branch] gh/malfet/607/orig -> origin/gh/malfet/607/orig 2025-12-04T08:57:44.3222624Z * [new branch] gh/malfet/608/base -> origin/gh/malfet/608/base 2025-12-04T08:57:44.3223846Z * [new branch] gh/malfet/608/head -> origin/gh/malfet/608/head 2025-12-04T08:57:44.3225472Z * [new branch] gh/malfet/608/orig -> origin/gh/malfet/608/orig 2025-12-04T08:57:44.3227800Z * [new branch] gh/malfet/609/base -> origin/gh/malfet/609/base 2025-12-04T08:57:44.3228986Z * [new branch] gh/malfet/609/head -> origin/gh/malfet/609/head 2025-12-04T08:57:44.3230107Z * [new branch] gh/malfet/609/orig -> origin/gh/malfet/609/orig 2025-12-04T08:57:44.3231646Z * [new branch] gh/malfet/610/base -> origin/gh/malfet/610/base 2025-12-04T08:57:44.3232950Z * [new branch] gh/malfet/610/head -> origin/gh/malfet/610/head 2025-12-04T08:57:44.3234016Z * [new branch] gh/malfet/610/orig -> origin/gh/malfet/610/orig 2025-12-04T08:57:44.3235565Z * [new branch] gh/malfet/611/base -> origin/gh/malfet/611/base 2025-12-04T08:57:44.3236676Z * [new branch] gh/malfet/611/head -> origin/gh/malfet/611/head 2025-12-04T08:57:44.3237836Z * [new branch] gh/malfet/611/orig -> origin/gh/malfet/611/orig 2025-12-04T08:57:44.3239486Z * [new branch] gh/malfet/612/base -> origin/gh/malfet/612/base 2025-12-04T08:57:44.3240598Z * [new branch] gh/malfet/612/head -> origin/gh/malfet/612/head 2025-12-04T08:57:44.3241724Z * [new branch] gh/malfet/612/orig -> origin/gh/malfet/612/orig 2025-12-04T08:57:44.3243303Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-12-04T08:57:44.3244397Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-12-04T08:57:44.3246153Z * [new branch] gh/manuelcandales/11/base -> origin/gh/manuelcandales/11/base 2025-12-04T08:57:44.3247315Z * [new branch] gh/manuelcandales/11/head -> origin/gh/manuelcandales/11/head 2025-12-04T08:57:44.3248402Z * [new branch] gh/manuelcandales/11/orig -> origin/gh/manuelcandales/11/orig 2025-12-04T08:57:44.3250424Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-12-04T08:57:44.3252215Z * [new branch] gh/masnesral/1/base -> origin/gh/masnesral/1/base 2025-12-04T08:57:44.3253311Z * [new branch] gh/masnesral/1/head -> origin/gh/masnesral/1/head 2025-12-04T08:57:44.3254392Z * [new branch] gh/masnesral/1/orig -> origin/gh/masnesral/1/orig 2025-12-04T08:57:44.3256280Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-12-04T08:57:44.3257467Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-12-04T08:57:44.3258753Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-12-04T08:57:44.3259873Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-12-04T08:57:44.3261170Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-12-04T08:57:44.3262267Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-12-04T08:57:44.3263583Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-12-04T08:57:44.3264655Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-12-04T08:57:44.3265940Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-12-04T08:57:44.3266955Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-12-04T08:57:44.3268717Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-12-04T08:57:44.3269771Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-12-04T08:57:44.3271054Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-12-04T08:57:44.3272158Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-12-04T08:57:44.3274021Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-12-04T08:57:44.3275247Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-12-04T08:57:44.3277103Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-12-04T08:57:44.3278598Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-12-04T08:57:44.3280205Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-12-04T08:57:44.3281157Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-12-04T08:57:44.3282679Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-12-04T08:57:44.3283717Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-12-04T08:57:44.3285191Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-12-04T08:57:44.3286260Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-12-04T08:57:44.3287748Z * [new branch] gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base 2025-12-04T08:57:44.3288892Z * [new branch] gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head 2025-12-04T08:57:44.3290044Z * [new branch] gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig 2025-12-04T08:57:44.3291585Z * [new branch] gh/mikaylagawarecki/341/base -> origin/gh/mikaylagawarecki/341/base 2025-12-04T08:57:44.3292640Z * [new branch] gh/mikaylagawarecki/341/head -> origin/gh/mikaylagawarecki/341/head 2025-12-04T08:57:44.3293722Z * [new branch] gh/mikaylagawarecki/341/orig -> origin/gh/mikaylagawarecki/341/orig 2025-12-04T08:57:44.3295431Z * [new branch] gh/mikaylagawarecki/342/base -> origin/gh/mikaylagawarecki/342/base 2025-12-04T08:57:44.3296505Z * [new branch] gh/mikaylagawarecki/342/head -> origin/gh/mikaylagawarecki/342/head 2025-12-04T08:57:44.3297748Z * [new branch] gh/mikaylagawarecki/342/orig -> origin/gh/mikaylagawarecki/342/orig 2025-12-04T08:57:44.3299287Z * [new branch] gh/mikaylagawarecki/345/base -> origin/gh/mikaylagawarecki/345/base 2025-12-04T08:57:44.3300321Z * [new branch] gh/mikaylagawarecki/345/head -> origin/gh/mikaylagawarecki/345/head 2025-12-04T08:57:44.3301482Z * [new branch] gh/mikaylagawarecki/345/orig -> origin/gh/mikaylagawarecki/345/orig 2025-12-04T08:57:44.3303057Z * [new branch] gh/mikaylagawarecki/346/base -> origin/gh/mikaylagawarecki/346/base 2025-12-04T08:57:44.3304150Z * [new branch] gh/mikaylagawarecki/346/head -> origin/gh/mikaylagawarecki/346/head 2025-12-04T08:57:44.3305411Z * [new branch] gh/mikaylagawarecki/346/orig -> origin/gh/mikaylagawarecki/346/orig 2025-12-04T08:57:44.3306937Z * [new branch] gh/mikaylagawarecki/347/base -> origin/gh/mikaylagawarecki/347/base 2025-12-04T08:57:44.3308280Z * [new branch] gh/mikaylagawarecki/347/head -> origin/gh/mikaylagawarecki/347/head 2025-12-04T08:57:44.3309490Z * [new branch] gh/mikaylagawarecki/347/orig -> origin/gh/mikaylagawarecki/347/orig 2025-12-04T08:57:44.3311139Z * [new branch] gh/mikaylagawarecki/350/base -> origin/gh/mikaylagawarecki/350/base 2025-12-04T08:57:44.3312241Z * [new branch] gh/mikaylagawarecki/350/head -> origin/gh/mikaylagawarecki/350/head 2025-12-04T08:57:44.3313479Z * [new branch] gh/mikaylagawarecki/350/orig -> origin/gh/mikaylagawarecki/350/orig 2025-12-04T08:57:44.3315913Z * [new branch] gh/mikaylagawarecki/351/base -> origin/gh/mikaylagawarecki/351/base 2025-12-04T08:57:44.3317148Z * [new branch] gh/mikaylagawarecki/351/head -> origin/gh/mikaylagawarecki/351/head 2025-12-04T08:57:44.3318290Z * [new branch] gh/mikaylagawarecki/351/orig -> origin/gh/mikaylagawarecki/351/orig 2025-12-04T08:57:44.3320087Z * [new branch] gh/mikaylagawarecki/352/base -> origin/gh/mikaylagawarecki/352/base 2025-12-04T08:57:44.3321362Z * [new branch] gh/mikaylagawarecki/352/head -> origin/gh/mikaylagawarecki/352/head 2025-12-04T08:57:44.3322966Z * [new branch] gh/mikaylagawarecki/352/orig -> origin/gh/mikaylagawarecki/352/orig 2025-12-04T08:57:44.3328441Z * [new branch] gh/mikaylagawarecki/353/base -> origin/gh/mikaylagawarecki/353/base 2025-12-04T08:57:44.3329837Z * [new branch] gh/mikaylagawarecki/353/head -> origin/gh/mikaylagawarecki/353/head 2025-12-04T08:57:44.3331046Z * [new branch] gh/mikaylagawarecki/353/orig -> origin/gh/mikaylagawarecki/353/orig 2025-12-04T08:57:44.3332947Z * [new branch] gh/mikaylagawarecki/354/base -> origin/gh/mikaylagawarecki/354/base 2025-12-04T08:57:44.3334105Z * [new branch] gh/mikaylagawarecki/354/head -> origin/gh/mikaylagawarecki/354/head 2025-12-04T08:57:44.3335856Z * [new branch] gh/mikaylagawarecki/354/orig -> origin/gh/mikaylagawarecki/354/orig 2025-12-04T08:57:44.3337886Z * [new branch] gh/mikaylagawarecki/356/base -> origin/gh/mikaylagawarecki/356/base 2025-12-04T08:57:44.3339043Z * [new branch] gh/mikaylagawarecki/356/head -> origin/gh/mikaylagawarecki/356/head 2025-12-04T08:57:44.3340147Z * [new branch] gh/mikaylagawarecki/356/orig -> origin/gh/mikaylagawarecki/356/orig 2025-12-04T08:57:44.3342017Z * [new branch] gh/mikaylagawarecki/357/base -> origin/gh/mikaylagawarecki/357/base 2025-12-04T08:57:44.3343184Z * [new branch] gh/mikaylagawarecki/357/head -> origin/gh/mikaylagawarecki/357/head 2025-12-04T08:57:44.3344274Z * [new branch] gh/mikaylagawarecki/357/orig -> origin/gh/mikaylagawarecki/357/orig 2025-12-04T08:57:44.3345911Z * [new branch] gh/mikaylagawarecki/359/base -> origin/gh/mikaylagawarecki/359/base 2025-12-04T08:57:44.3347587Z * [new branch] gh/mikaylagawarecki/359/head -> origin/gh/mikaylagawarecki/359/head 2025-12-04T08:57:44.3349021Z * [new branch] gh/mikaylagawarecki/359/orig -> origin/gh/mikaylagawarecki/359/orig 2025-12-04T08:57:44.3350557Z * [new branch] gh/mikaylagawarecki/360/base -> origin/gh/mikaylagawarecki/360/base 2025-12-04T08:57:44.3351749Z * [new branch] gh/mikaylagawarecki/360/head -> origin/gh/mikaylagawarecki/360/head 2025-12-04T08:57:44.3353018Z * [new branch] gh/mikaylagawarecki/360/orig -> origin/gh/mikaylagawarecki/360/orig 2025-12-04T08:57:44.3354610Z * [new branch] gh/mikaylagawarecki/361/base -> origin/gh/mikaylagawarecki/361/base 2025-12-04T08:57:44.3355756Z * [new branch] gh/mikaylagawarecki/361/head -> origin/gh/mikaylagawarecki/361/head 2025-12-04T08:57:44.3356865Z * [new branch] gh/mikaylagawarecki/361/orig -> origin/gh/mikaylagawarecki/361/orig 2025-12-04T08:57:44.3358547Z * [new branch] gh/mikaylagawarecki/362/base -> origin/gh/mikaylagawarecki/362/base 2025-12-04T08:57:44.3360003Z * [new branch] gh/mikaylagawarecki/362/head -> origin/gh/mikaylagawarecki/362/head 2025-12-04T08:57:44.3361110Z * [new branch] gh/mikaylagawarecki/362/orig -> origin/gh/mikaylagawarecki/362/orig 2025-12-04T08:57:44.3362958Z * [new branch] gh/mikaylagawarecki/363/base -> origin/gh/mikaylagawarecki/363/base 2025-12-04T08:57:44.3364266Z * [new branch] gh/mikaylagawarecki/363/head -> origin/gh/mikaylagawarecki/363/head 2025-12-04T08:57:44.3365942Z * [new branch] gh/mikaylagawarecki/363/orig -> origin/gh/mikaylagawarecki/363/orig 2025-12-04T08:57:44.3367832Z * [new branch] gh/mikaylagawarecki/364/base -> origin/gh/mikaylagawarecki/364/base 2025-12-04T08:57:44.3368929Z * [new branch] gh/mikaylagawarecki/364/head -> origin/gh/mikaylagawarecki/364/head 2025-12-04T08:57:44.3370203Z * [new branch] gh/mikaylagawarecki/364/orig -> origin/gh/mikaylagawarecki/364/orig 2025-12-04T08:57:44.3371911Z * [new branch] gh/mikaylagawarecki/365/base -> origin/gh/mikaylagawarecki/365/base 2025-12-04T08:57:44.3372985Z * [new branch] gh/mikaylagawarecki/365/head -> origin/gh/mikaylagawarecki/365/head 2025-12-04T08:57:44.3374367Z * [new branch] gh/mikaylagawarecki/365/orig -> origin/gh/mikaylagawarecki/365/orig 2025-12-04T08:57:44.3375854Z * [new branch] gh/mikaylagawarecki/366/base -> origin/gh/mikaylagawarecki/366/base 2025-12-04T08:57:44.3376825Z * [new branch] gh/mikaylagawarecki/366/head -> origin/gh/mikaylagawarecki/366/head 2025-12-04T08:57:44.3378018Z * [new branch] gh/mikaylagawarecki/366/orig -> origin/gh/mikaylagawarecki/366/orig 2025-12-04T08:57:44.3379528Z * [new branch] gh/mikaylagawarecki/367/base -> origin/gh/mikaylagawarecki/367/base 2025-12-04T08:57:44.3380634Z * [new branch] gh/mikaylagawarecki/367/head -> origin/gh/mikaylagawarecki/367/head 2025-12-04T08:57:44.3381855Z * [new branch] gh/mikaylagawarecki/367/orig -> origin/gh/mikaylagawarecki/367/orig 2025-12-04T08:57:44.3383457Z * [new branch] gh/mikaylagawarecki/368/base -> origin/gh/mikaylagawarecki/368/base 2025-12-04T08:57:44.3384564Z * [new branch] gh/mikaylagawarecki/368/head -> origin/gh/mikaylagawarecki/368/head 2025-12-04T08:57:44.3385783Z * [new branch] gh/mikaylagawarecki/368/orig -> origin/gh/mikaylagawarecki/368/orig 2025-12-04T08:57:44.3387360Z * [new branch] gh/mikaylagawarecki/369/base -> origin/gh/mikaylagawarecki/369/base 2025-12-04T08:57:44.3388785Z * [new branch] gh/mikaylagawarecki/369/head -> origin/gh/mikaylagawarecki/369/head 2025-12-04T08:57:44.3389904Z * [new branch] gh/mikaylagawarecki/369/orig -> origin/gh/mikaylagawarecki/369/orig 2025-12-04T08:57:44.3391559Z * [new branch] gh/mikaylagawarecki/370/base -> origin/gh/mikaylagawarecki/370/base 2025-12-04T08:57:44.3392705Z * [new branch] gh/mikaylagawarecki/370/head -> origin/gh/mikaylagawarecki/370/head 2025-12-04T08:57:44.3393879Z * [new branch] gh/mikaylagawarecki/370/orig -> origin/gh/mikaylagawarecki/370/orig 2025-12-04T08:57:44.3395471Z * [new branch] gh/mikaylagawarecki/371/base -> origin/gh/mikaylagawarecki/371/base 2025-12-04T08:57:44.3396544Z * [new branch] gh/mikaylagawarecki/371/head -> origin/gh/mikaylagawarecki/371/head 2025-12-04T08:57:44.3397637Z * [new branch] gh/mikaylagawarecki/371/orig -> origin/gh/mikaylagawarecki/371/orig 2025-12-04T08:57:44.3399176Z * [new branch] gh/mikaylagawarecki/372/base -> origin/gh/mikaylagawarecki/372/base 2025-12-04T08:57:44.3400527Z * [new branch] gh/mikaylagawarecki/372/head -> origin/gh/mikaylagawarecki/372/head 2025-12-04T08:57:44.3401749Z * [new branch] gh/mikaylagawarecki/372/orig -> origin/gh/mikaylagawarecki/372/orig 2025-12-04T08:57:44.3403163Z * [new branch] gh/mikaylagawarecki/373/base -> origin/gh/mikaylagawarecki/373/base 2025-12-04T08:57:44.3404245Z * [new branch] gh/mikaylagawarecki/373/head -> origin/gh/mikaylagawarecki/373/head 2025-12-04T08:57:44.3405336Z * [new branch] gh/mikaylagawarecki/373/orig -> origin/gh/mikaylagawarecki/373/orig 2025-12-04T08:57:44.3407314Z * [new branch] gh/mikaylagawarecki/374/base -> origin/gh/mikaylagawarecki/374/base 2025-12-04T08:57:44.3408421Z * [new branch] gh/mikaylagawarecki/374/head -> origin/gh/mikaylagawarecki/374/head 2025-12-04T08:57:44.3409493Z * [new branch] gh/mikaylagawarecki/374/orig -> origin/gh/mikaylagawarecki/374/orig 2025-12-04T08:57:44.3411008Z * [new branch] gh/mikaylagawarecki/375/base -> origin/gh/mikaylagawarecki/375/base 2025-12-04T08:57:44.3412209Z * [new branch] gh/mikaylagawarecki/375/head -> origin/gh/mikaylagawarecki/375/head 2025-12-04T08:57:44.3413327Z * [new branch] gh/mikaylagawarecki/375/orig -> origin/gh/mikaylagawarecki/375/orig 2025-12-04T08:57:44.3414827Z * [new branch] gh/mikaylagawarecki/376/base -> origin/gh/mikaylagawarecki/376/base 2025-12-04T08:57:44.3415990Z * [new branch] gh/mikaylagawarecki/376/head -> origin/gh/mikaylagawarecki/376/head 2025-12-04T08:57:44.3417317Z * [new branch] gh/mikaylagawarecki/376/orig -> origin/gh/mikaylagawarecki/376/orig 2025-12-04T08:57:44.3419244Z * [new branch] gh/mikaylagawarecki/377/base -> origin/gh/mikaylagawarecki/377/base 2025-12-04T08:57:44.3420390Z * [new branch] gh/mikaylagawarecki/377/head -> origin/gh/mikaylagawarecki/377/head 2025-12-04T08:57:44.3421510Z * [new branch] gh/mikaylagawarecki/377/orig -> origin/gh/mikaylagawarecki/377/orig 2025-12-04T08:57:44.3423011Z * [new branch] gh/mikaylagawarecki/378/base -> origin/gh/mikaylagawarecki/378/base 2025-12-04T08:57:44.3424606Z * [new branch] gh/mikaylagawarecki/378/head -> origin/gh/mikaylagawarecki/378/head 2025-12-04T08:57:44.3425750Z * [new branch] gh/mikaylagawarecki/378/orig -> origin/gh/mikaylagawarecki/378/orig 2025-12-04T08:57:44.3427422Z * [new branch] gh/mikaylagawarecki/379/base -> origin/gh/mikaylagawarecki/379/base 2025-12-04T08:57:44.3429088Z * [new branch] gh/mikaylagawarecki/379/head -> origin/gh/mikaylagawarecki/379/head 2025-12-04T08:57:44.3430206Z * [new branch] gh/mikaylagawarecki/379/orig -> origin/gh/mikaylagawarecki/379/orig 2025-12-04T08:57:44.3431625Z * [new branch] gh/mikaylagawarecki/380/base -> origin/gh/mikaylagawarecki/380/base 2025-12-04T08:57:44.3432753Z * [new branch] gh/mikaylagawarecki/380/head -> origin/gh/mikaylagawarecki/380/head 2025-12-04T08:57:44.3433962Z * [new branch] gh/mikaylagawarecki/380/orig -> origin/gh/mikaylagawarecki/380/orig 2025-12-04T08:57:44.3435334Z * [new branch] gh/mikaylagawarecki/381/base -> origin/gh/mikaylagawarecki/381/base 2025-12-04T08:57:44.3436468Z * [new branch] gh/mikaylagawarecki/381/head -> origin/gh/mikaylagawarecki/381/head 2025-12-04T08:57:44.3437584Z * [new branch] gh/mikaylagawarecki/381/orig -> origin/gh/mikaylagawarecki/381/orig 2025-12-04T08:57:44.3439143Z * [new branch] gh/mikaylagawarecki/382/base -> origin/gh/mikaylagawarecki/382/base 2025-12-04T08:57:44.3440238Z * [new branch] gh/mikaylagawarecki/382/head -> origin/gh/mikaylagawarecki/382/head 2025-12-04T08:57:44.3441311Z * [new branch] gh/mikaylagawarecki/382/orig -> origin/gh/mikaylagawarecki/382/orig 2025-12-04T08:57:44.3442916Z * [new branch] gh/mikaylagawarecki/383/base -> origin/gh/mikaylagawarecki/383/base 2025-12-04T08:57:44.3443985Z * [new branch] gh/mikaylagawarecki/383/head -> origin/gh/mikaylagawarecki/383/head 2025-12-04T08:57:44.3445169Z * [new branch] gh/mikaylagawarecki/383/orig -> origin/gh/mikaylagawarecki/383/orig 2025-12-04T08:57:44.3446643Z * [new branch] gh/mikaylagawarecki/384/base -> origin/gh/mikaylagawarecki/384/base 2025-12-04T08:57:44.3447741Z * [new branch] gh/mikaylagawarecki/384/head -> origin/gh/mikaylagawarecki/384/head 2025-12-04T08:57:44.3449074Z * [new branch] gh/mikaylagawarecki/384/orig -> origin/gh/mikaylagawarecki/384/orig 2025-12-04T08:57:44.3450637Z * [new branch] gh/mikaylagawarecki/385/base -> origin/gh/mikaylagawarecki/385/base 2025-12-04T08:57:44.3451808Z * [new branch] gh/mikaylagawarecki/385/head -> origin/gh/mikaylagawarecki/385/head 2025-12-04T08:57:44.3452932Z * [new branch] gh/mikaylagawarecki/385/orig -> origin/gh/mikaylagawarecki/385/orig 2025-12-04T08:57:44.3454561Z * [new branch] gh/mikaylagawarecki/386/base -> origin/gh/mikaylagawarecki/386/base 2025-12-04T08:57:44.3455606Z * [new branch] gh/mikaylagawarecki/386/head -> origin/gh/mikaylagawarecki/386/head 2025-12-04T08:57:44.3456707Z * [new branch] gh/mikaylagawarecki/386/orig -> origin/gh/mikaylagawarecki/386/orig 2025-12-04T08:57:44.3458212Z * [new branch] gh/mikaylagawarecki/387/base -> origin/gh/mikaylagawarecki/387/base 2025-12-04T08:57:44.3459454Z * [new branch] gh/mikaylagawarecki/387/head -> origin/gh/mikaylagawarecki/387/head 2025-12-04T08:57:44.3460468Z * [new branch] gh/mikaylagawarecki/387/orig -> origin/gh/mikaylagawarecki/387/orig 2025-12-04T08:57:44.3461849Z * [new branch] gh/mikaylagawarecki/388/base -> origin/gh/mikaylagawarecki/388/base 2025-12-04T08:57:44.3462937Z * [new branch] gh/mikaylagawarecki/388/head -> origin/gh/mikaylagawarecki/388/head 2025-12-04T08:57:44.3464113Z * [new branch] gh/mikaylagawarecki/388/orig -> origin/gh/mikaylagawarecki/388/orig 2025-12-04T08:57:44.3465636Z * [new branch] gh/mikaylagawarecki/389/base -> origin/gh/mikaylagawarecki/389/base 2025-12-04T08:57:44.3466744Z * [new branch] gh/mikaylagawarecki/389/head -> origin/gh/mikaylagawarecki/389/head 2025-12-04T08:57:44.3468114Z * [new branch] gh/mikaylagawarecki/389/orig -> origin/gh/mikaylagawarecki/389/orig 2025-12-04T08:57:44.3469763Z * [new branch] gh/mikaylagawarecki/390/base -> origin/gh/mikaylagawarecki/390/base 2025-12-04T08:57:44.3470820Z * [new branch] gh/mikaylagawarecki/390/head -> origin/gh/mikaylagawarecki/390/head 2025-12-04T08:57:44.3471965Z * [new branch] gh/mikaylagawarecki/390/orig -> origin/gh/mikaylagawarecki/390/orig 2025-12-04T08:57:44.3473627Z * [new branch] gh/mikaylagawarecki/391/base -> origin/gh/mikaylagawarecki/391/base 2025-12-04T08:57:44.3474874Z * [new branch] gh/mikaylagawarecki/391/head -> origin/gh/mikaylagawarecki/391/head 2025-12-04T08:57:44.3476021Z * [new branch] gh/mikaylagawarecki/391/orig -> origin/gh/mikaylagawarecki/391/orig 2025-12-04T08:57:44.3478144Z * [new branch] gh/mikaylagawarecki/392/base -> origin/gh/mikaylagawarecki/392/base 2025-12-04T08:57:44.3479322Z * [new branch] gh/mikaylagawarecki/392/head -> origin/gh/mikaylagawarecki/392/head 2025-12-04T08:57:44.3480641Z * [new branch] gh/mikaylagawarecki/392/orig -> origin/gh/mikaylagawarecki/392/orig 2025-12-04T08:57:44.3482397Z * [new branch] gh/mlazos/41/base -> origin/gh/mlazos/41/base 2025-12-04T08:57:44.3483469Z * [new branch] gh/mlazos/41/head -> origin/gh/mlazos/41/head 2025-12-04T08:57:44.3484546Z * [new branch] gh/mlazos/41/orig -> origin/gh/mlazos/41/orig 2025-12-04T08:57:44.3486109Z * [new branch] gh/mlazos/42/base -> origin/gh/mlazos/42/base 2025-12-04T08:57:44.3487118Z * [new branch] gh/mlazos/42/head -> origin/gh/mlazos/42/head 2025-12-04T08:57:44.3488260Z * [new branch] gh/mlazos/42/orig -> origin/gh/mlazos/42/orig 2025-12-04T08:57:44.3489520Z * [new branch] gh/mlazos/43/base -> origin/gh/mlazos/43/base 2025-12-04T08:57:44.3490631Z * [new branch] gh/mlazos/43/head -> origin/gh/mlazos/43/head 2025-12-04T08:57:44.3491747Z * [new branch] gh/mlazos/43/orig -> origin/gh/mlazos/43/orig 2025-12-04T08:57:44.3493074Z * [new branch] gh/mlazos/44/base -> origin/gh/mlazos/44/base 2025-12-04T08:57:44.3494259Z * [new branch] gh/mlazos/44/head -> origin/gh/mlazos/44/head 2025-12-04T08:57:44.3495341Z * [new branch] gh/mlazos/44/orig -> origin/gh/mlazos/44/orig 2025-12-04T08:57:44.3496802Z * [new branch] gh/mlazos/47/base -> origin/gh/mlazos/47/base 2025-12-04T08:57:44.3497913Z * [new branch] gh/mlazos/47/head -> origin/gh/mlazos/47/head 2025-12-04T08:57:44.3498997Z * [new branch] gh/mlazos/47/orig -> origin/gh/mlazos/47/orig 2025-12-04T08:57:44.3500328Z * [new branch] gh/mlazos/48/base -> origin/gh/mlazos/48/base 2025-12-04T08:57:44.3501394Z * [new branch] gh/mlazos/48/head -> origin/gh/mlazos/48/head 2025-12-04T08:57:44.3502608Z * [new branch] gh/mlazos/48/orig -> origin/gh/mlazos/48/orig 2025-12-04T08:57:44.3503921Z * [new branch] gh/mlazos/49/base -> origin/gh/mlazos/49/base 2025-12-04T08:57:44.3505026Z * [new branch] gh/mlazos/49/head -> origin/gh/mlazos/49/head 2025-12-04T08:57:44.3506103Z * [new branch] gh/mlazos/49/orig -> origin/gh/mlazos/49/orig 2025-12-04T08:57:44.3507911Z * [new branch] gh/mlazos/50/base -> origin/gh/mlazos/50/base 2025-12-04T08:57:44.3508970Z * [new branch] gh/mlazos/50/head -> origin/gh/mlazos/50/head 2025-12-04T08:57:44.3510087Z * [new branch] gh/mlazos/50/orig -> origin/gh/mlazos/50/orig 2025-12-04T08:57:44.3511529Z * [new branch] gh/mlazos/51/base -> origin/gh/mlazos/51/base 2025-12-04T08:57:44.3512648Z * [new branch] gh/mlazos/51/head -> origin/gh/mlazos/51/head 2025-12-04T08:57:44.3514104Z * [new branch] gh/mlazos/51/orig -> origin/gh/mlazos/51/orig 2025-12-04T08:57:44.3515419Z * [new branch] gh/mlazos/52/base -> origin/gh/mlazos/52/base 2025-12-04T08:57:44.3516512Z * [new branch] gh/mlazos/52/head -> origin/gh/mlazos/52/head 2025-12-04T08:57:44.3517664Z * [new branch] gh/mlazos/52/orig -> origin/gh/mlazos/52/orig 2025-12-04T08:57:44.3519154Z * [new branch] gh/mlazos/53/base -> origin/gh/mlazos/53/base 2025-12-04T08:57:44.3520400Z * [new branch] gh/mlazos/53/head -> origin/gh/mlazos/53/head 2025-12-04T08:57:44.3521475Z * [new branch] gh/mlazos/53/orig -> origin/gh/mlazos/53/orig 2025-12-04T08:57:44.3522950Z * [new branch] gh/mlazos/54/base -> origin/gh/mlazos/54/base 2025-12-04T08:57:44.3524559Z * [new branch] gh/mlazos/54/head -> origin/gh/mlazos/54/head 2025-12-04T08:57:44.3525707Z * [new branch] gh/mlazos/54/orig -> origin/gh/mlazos/54/orig 2025-12-04T08:57:44.3527114Z * [new branch] gh/mlazos/55/base -> origin/gh/mlazos/55/base 2025-12-04T08:57:44.3528258Z * [new branch] gh/mlazos/55/head -> origin/gh/mlazos/55/head 2025-12-04T08:57:44.3529386Z * [new branch] gh/mlazos/55/orig -> origin/gh/mlazos/55/orig 2025-12-04T08:57:44.3531274Z * [new branch] gh/mlazos/56/base -> origin/gh/mlazos/56/base 2025-12-04T08:57:44.3532427Z * [new branch] gh/mlazos/56/head -> origin/gh/mlazos/56/head 2025-12-04T08:57:44.3533532Z * [new branch] gh/mlazos/56/orig -> origin/gh/mlazos/56/orig 2025-12-04T08:57:44.3535010Z * [new branch] gh/mlazos/57/base -> origin/gh/mlazos/57/base 2025-12-04T08:57:44.3536122Z * [new branch] gh/mlazos/57/head -> origin/gh/mlazos/57/head 2025-12-04T08:57:44.3537308Z * [new branch] gh/mlazos/57/orig -> origin/gh/mlazos/57/orig 2025-12-04T08:57:44.3538753Z * [new branch] gh/mlazos/58/base -> origin/gh/mlazos/58/base 2025-12-04T08:57:44.3539993Z * [new branch] gh/mlazos/58/head -> origin/gh/mlazos/58/head 2025-12-04T08:57:44.3541106Z * [new branch] gh/mlazos/58/orig -> origin/gh/mlazos/58/orig 2025-12-04T08:57:44.3542514Z * [new branch] gh/mlazos/59/base -> origin/gh/mlazos/59/base 2025-12-04T08:57:44.3543605Z * [new branch] gh/mlazos/59/head -> origin/gh/mlazos/59/head 2025-12-04T08:57:44.3544635Z * [new branch] gh/mlazos/59/orig -> origin/gh/mlazos/59/orig 2025-12-04T08:57:44.3546089Z * [new branch] gh/mlazos/60/base -> origin/gh/mlazos/60/base 2025-12-04T08:57:44.3547202Z * [new branch] gh/mlazos/60/head -> origin/gh/mlazos/60/head 2025-12-04T08:57:44.3548902Z * [new branch] gh/mlazos/60/orig -> origin/gh/mlazos/60/orig 2025-12-04T08:57:44.3550628Z * [new branch] gh/mlazos/61/base -> origin/gh/mlazos/61/base 2025-12-04T08:57:44.3551802Z * [new branch] gh/mlazos/61/head -> origin/gh/mlazos/61/head 2025-12-04T08:57:44.3552923Z * [new branch] gh/mlazos/61/orig -> origin/gh/mlazos/61/orig 2025-12-04T08:57:44.3554459Z * [new branch] gh/mlazos/62/base -> origin/gh/mlazos/62/base 2025-12-04T08:57:44.3555682Z * [new branch] gh/mlazos/62/head -> origin/gh/mlazos/62/head 2025-12-04T08:57:44.3556818Z * [new branch] gh/mlazos/62/orig -> origin/gh/mlazos/62/orig 2025-12-04T08:57:44.3558395Z * [new branch] gh/mlazos/63/base -> origin/gh/mlazos/63/base 2025-12-04T08:57:44.3559717Z * [new branch] gh/mlazos/63/head -> origin/gh/mlazos/63/head 2025-12-04T08:57:44.3560907Z * [new branch] gh/mlazos/63/orig -> origin/gh/mlazos/63/orig 2025-12-04T08:57:44.3562316Z * [new branch] gh/mlazos/64/base -> origin/gh/mlazos/64/base 2025-12-04T08:57:44.3563423Z * [new branch] gh/mlazos/64/head -> origin/gh/mlazos/64/head 2025-12-04T08:57:44.3564478Z * [new branch] gh/mlazos/64/orig -> origin/gh/mlazos/64/orig 2025-12-04T08:57:44.3565964Z * [new branch] gh/mlazos/65/base -> origin/gh/mlazos/65/base 2025-12-04T08:57:44.3567046Z * [new branch] gh/mlazos/65/head -> origin/gh/mlazos/65/head 2025-12-04T08:57:44.3568138Z * [new branch] gh/mlazos/65/orig -> origin/gh/mlazos/65/orig 2025-12-04T08:57:44.3569588Z * [new branch] gh/mlazos/66/base -> origin/gh/mlazos/66/base 2025-12-04T08:57:44.3570802Z * [new branch] gh/mlazos/66/head -> origin/gh/mlazos/66/head 2025-12-04T08:57:44.3571891Z * [new branch] gh/mlazos/66/orig -> origin/gh/mlazos/66/orig 2025-12-04T08:57:44.3573361Z * [new branch] gh/mlazos/67/base -> origin/gh/mlazos/67/base 2025-12-04T08:57:44.3574420Z * [new branch] gh/mlazos/67/head -> origin/gh/mlazos/67/head 2025-12-04T08:57:44.3575499Z * [new branch] gh/mlazos/67/orig -> origin/gh/mlazos/67/orig 2025-12-04T08:57:44.3576946Z * [new branch] gh/mlazos/68/base -> origin/gh/mlazos/68/base 2025-12-04T08:57:44.3578117Z * [new branch] gh/mlazos/68/head -> origin/gh/mlazos/68/head 2025-12-04T08:57:44.3579710Z * [new branch] gh/mlazos/68/orig -> origin/gh/mlazos/68/orig 2025-12-04T08:57:44.3581647Z * [new branch] gh/mlazos/69/base -> origin/gh/mlazos/69/base 2025-12-04T08:57:44.3582774Z * [new branch] gh/mlazos/69/head -> origin/gh/mlazos/69/head 2025-12-04T08:57:44.3585852Z * [new branch] gh/mlazos/69/orig -> origin/gh/mlazos/69/orig 2025-12-04T08:57:44.3586217Z * [new branch] gh/mlazos/70/base -> origin/gh/mlazos/70/base 2025-12-04T08:57:44.3587177Z * [new branch] gh/mlazos/70/head -> origin/gh/mlazos/70/head 2025-12-04T08:57:44.3588669Z * [new branch] gh/mlazos/70/orig -> origin/gh/mlazos/70/orig 2025-12-04T08:57:44.3590047Z * [new branch] gh/mlazos/71/base -> origin/gh/mlazos/71/base 2025-12-04T08:57:44.3591193Z * [new branch] gh/mlazos/71/head -> origin/gh/mlazos/71/head 2025-12-04T08:57:44.3592290Z * [new branch] gh/mlazos/71/orig -> origin/gh/mlazos/71/orig 2025-12-04T08:57:44.3593741Z * [new branch] gh/mlazos/72/base -> origin/gh/mlazos/72/base 2025-12-04T08:57:44.3594903Z * [new branch] gh/mlazos/72/head -> origin/gh/mlazos/72/head 2025-12-04T08:57:44.3596090Z * [new branch] gh/mlazos/72/orig -> origin/gh/mlazos/72/orig 2025-12-04T08:57:44.3597668Z * [new branch] gh/mlazos/73/base -> origin/gh/mlazos/73/base 2025-12-04T08:57:44.3598798Z * [new branch] gh/mlazos/73/head -> origin/gh/mlazos/73/head 2025-12-04T08:57:44.3600080Z * [new branch] gh/mlazos/73/orig -> origin/gh/mlazos/73/orig 2025-12-04T08:57:44.3601969Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-12-04T08:57:44.3603131Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-12-04T08:57:44.3605053Z * [new branch] gh/muchulee8/73/base -> origin/gh/muchulee8/73/base 2025-12-04T08:57:44.3606286Z * [new branch] gh/muchulee8/73/head -> origin/gh/muchulee8/73/head 2025-12-04T08:57:44.3607474Z * [new branch] gh/muchulee8/73/orig -> origin/gh/muchulee8/73/orig 2025-12-04T08:57:44.3609305Z * [new branch] gh/naveenthangudu/1/base -> origin/gh/naveenthangudu/1/base 2025-12-04T08:57:44.3610432Z * [new branch] gh/naveenthangudu/1/head -> origin/gh/naveenthangudu/1/head 2025-12-04T08:57:44.3611634Z * [new branch] gh/naveenthangudu/1/orig -> origin/gh/naveenthangudu/1/orig 2025-12-04T08:57:44.3613082Z * [new branch] gh/naveenthangudu/2/base -> origin/gh/naveenthangudu/2/base 2025-12-04T08:57:44.3614186Z * [new branch] gh/naveenthangudu/2/head -> origin/gh/naveenthangudu/2/head 2025-12-04T08:57:44.3615325Z * [new branch] gh/naveenthangudu/2/orig -> origin/gh/naveenthangudu/2/orig 2025-12-04T08:57:44.3616778Z * [new branch] gh/naveenthangudu/3/base -> origin/gh/naveenthangudu/3/base 2025-12-04T08:57:44.3617885Z * [new branch] gh/naveenthangudu/3/head -> origin/gh/naveenthangudu/3/head 2025-12-04T08:57:44.3619040Z * [new branch] gh/naveenthangudu/3/orig -> origin/gh/naveenthangudu/3/orig 2025-12-04T08:57:44.3620507Z * [new branch] gh/naveenthangudu/4/base -> origin/gh/naveenthangudu/4/base 2025-12-04T08:57:44.3621561Z * [new branch] gh/naveenthangudu/4/head -> origin/gh/naveenthangudu/4/head 2025-12-04T08:57:44.3622778Z * [new branch] gh/naveenthangudu/4/orig -> origin/gh/naveenthangudu/4/orig 2025-12-04T08:57:44.3624676Z * [new branch] gh/naveenthangudu/5/base -> origin/gh/naveenthangudu/5/base 2025-12-04T08:57:44.3625827Z * [new branch] gh/naveenthangudu/5/head -> origin/gh/naveenthangudu/5/head 2025-12-04T08:57:44.3627227Z * [new branch] gh/naveenthangudu/5/orig -> origin/gh/naveenthangudu/5/orig 2025-12-04T08:57:44.3628926Z * [new branch] gh/naveenthangudu/6/base -> origin/gh/naveenthangudu/6/base 2025-12-04T08:57:44.3630029Z * [new branch] gh/naveenthangudu/6/head -> origin/gh/naveenthangudu/6/head 2025-12-04T08:57:44.3631103Z * [new branch] gh/naveenthangudu/6/orig -> origin/gh/naveenthangudu/6/orig 2025-12-04T08:57:44.3632729Z * [new branch] gh/naveenthangudu/7/base -> origin/gh/naveenthangudu/7/base 2025-12-04T08:57:44.3633835Z * [new branch] gh/naveenthangudu/7/head -> origin/gh/naveenthangudu/7/head 2025-12-04T08:57:44.3634918Z * [new branch] gh/naveenthangudu/7/orig -> origin/gh/naveenthangudu/7/orig 2025-12-04T08:57:44.3636288Z * [new branch] gh/naveenthangudu/8/base -> origin/gh/naveenthangudu/8/base 2025-12-04T08:57:44.3637444Z * [new branch] gh/naveenthangudu/8/head -> origin/gh/naveenthangudu/8/head 2025-12-04T08:57:44.3638669Z * [new branch] gh/naveenthangudu/8/orig -> origin/gh/naveenthangudu/8/orig 2025-12-04T08:57:44.3640212Z * [new branch] gh/naveenthangudu/9/base -> origin/gh/naveenthangudu/9/base 2025-12-04T08:57:44.3641316Z * [new branch] gh/naveenthangudu/9/head -> origin/gh/naveenthangudu/9/head 2025-12-04T08:57:44.3642536Z * [new branch] gh/naveenthangudu/9/orig -> origin/gh/naveenthangudu/9/orig 2025-12-04T08:57:44.3644108Z * [new branch] gh/nikitaved/1/base -> origin/gh/nikitaved/1/base 2025-12-04T08:57:44.3645185Z * [new branch] gh/nikitaved/1/head -> origin/gh/nikitaved/1/head 2025-12-04T08:57:44.3646607Z * [new branch] gh/nikitaved/1/orig -> origin/gh/nikitaved/1/orig 2025-12-04T08:57:44.3648154Z * [new branch] gh/nikitaved/10/base -> origin/gh/nikitaved/10/base 2025-12-04T08:57:44.3649294Z * [new branch] gh/nikitaved/10/head -> origin/gh/nikitaved/10/head 2025-12-04T08:57:44.3650385Z * [new branch] gh/nikitaved/10/orig -> origin/gh/nikitaved/10/orig 2025-12-04T08:57:44.3651767Z * [new branch] gh/nikitaved/11/base -> origin/gh/nikitaved/11/base 2025-12-04T08:57:44.3652927Z * [new branch] gh/nikitaved/11/head -> origin/gh/nikitaved/11/head 2025-12-04T08:57:44.3654003Z * [new branch] gh/nikitaved/11/orig -> origin/gh/nikitaved/11/orig 2025-12-04T08:57:44.3655407Z * [new branch] gh/nikitaved/12/base -> origin/gh/nikitaved/12/base 2025-12-04T08:57:44.3656534Z * [new branch] gh/nikitaved/12/head -> origin/gh/nikitaved/12/head 2025-12-04T08:57:44.3657689Z * [new branch] gh/nikitaved/12/orig -> origin/gh/nikitaved/12/orig 2025-12-04T08:57:44.3659139Z * [new branch] gh/nikitaved/13/base -> origin/gh/nikitaved/13/base 2025-12-04T08:57:44.3660314Z * [new branch] gh/nikitaved/13/head -> origin/gh/nikitaved/13/head 2025-12-04T08:57:44.3661548Z * [new branch] gh/nikitaved/13/orig -> origin/gh/nikitaved/13/orig 2025-12-04T08:57:44.3663034Z * [new branch] gh/nikitaved/14/base -> origin/gh/nikitaved/14/base 2025-12-04T08:57:44.3664100Z * [new branch] gh/nikitaved/14/head -> origin/gh/nikitaved/14/head 2025-12-04T08:57:44.3665230Z * [new branch] gh/nikitaved/14/orig -> origin/gh/nikitaved/14/orig 2025-12-04T08:57:44.3666581Z * [new branch] gh/nikitaved/15/base -> origin/gh/nikitaved/15/base 2025-12-04T08:57:44.3667994Z * [new branch] gh/nikitaved/15/head -> origin/gh/nikitaved/15/head 2025-12-04T08:57:44.3669146Z * [new branch] gh/nikitaved/15/orig -> origin/gh/nikitaved/15/orig 2025-12-04T08:57:44.3670651Z * [new branch] gh/nikitaved/16/base -> origin/gh/nikitaved/16/base 2025-12-04T08:57:44.3671718Z * [new branch] gh/nikitaved/16/head -> origin/gh/nikitaved/16/head 2025-12-04T08:57:44.3672826Z * [new branch] gh/nikitaved/16/orig -> origin/gh/nikitaved/16/orig 2025-12-04T08:57:44.3674382Z * [new branch] gh/nikitaved/2/base -> origin/gh/nikitaved/2/base 2025-12-04T08:57:44.3675534Z * [new branch] gh/nikitaved/2/head -> origin/gh/nikitaved/2/head 2025-12-04T08:57:44.3676736Z * [new branch] gh/nikitaved/2/orig -> origin/gh/nikitaved/2/orig 2025-12-04T08:57:44.3678105Z * [new branch] gh/nikitaved/4/base -> origin/gh/nikitaved/4/base 2025-12-04T08:57:44.3679281Z * [new branch] gh/nikitaved/4/head -> origin/gh/nikitaved/4/head 2025-12-04T08:57:44.3680511Z * [new branch] gh/nikitaved/4/orig -> origin/gh/nikitaved/4/orig 2025-12-04T08:57:44.3681982Z * [new branch] gh/nikitaved/5/base -> origin/gh/nikitaved/5/base 2025-12-04T08:57:44.3683527Z * [new branch] gh/nikitaved/5/head -> origin/gh/nikitaved/5/head 2025-12-04T08:57:44.3684649Z * [new branch] gh/nikitaved/5/orig -> origin/gh/nikitaved/5/orig 2025-12-04T08:57:44.3686083Z * [new branch] gh/nikitaved/6/base -> origin/gh/nikitaved/6/base 2025-12-04T08:57:44.3687240Z * [new branch] gh/nikitaved/6/head -> origin/gh/nikitaved/6/head 2025-12-04T08:57:44.3688262Z * [new branch] gh/nikitaved/6/orig -> origin/gh/nikitaved/6/orig 2025-12-04T08:57:44.3689730Z * [new branch] gh/nikitaved/8/base -> origin/gh/nikitaved/8/base 2025-12-04T08:57:44.3690872Z * [new branch] gh/nikitaved/8/head -> origin/gh/nikitaved/8/head 2025-12-04T08:57:44.3692022Z * [new branch] gh/nikitaved/8/orig -> origin/gh/nikitaved/8/orig 2025-12-04T08:57:44.3693455Z * [new branch] gh/nikitaved/9/base -> origin/gh/nikitaved/9/base 2025-12-04T08:57:44.3694565Z * [new branch] gh/nikitaved/9/head -> origin/gh/nikitaved/9/head 2025-12-04T08:57:44.3695642Z * [new branch] gh/nikitaved/9/orig -> origin/gh/nikitaved/9/orig 2025-12-04T08:57:44.3697355Z * [new branch] gh/oulgen/10/base -> origin/gh/oulgen/10/base 2025-12-04T08:57:44.3698433Z * [new branch] gh/oulgen/10/head -> origin/gh/oulgen/10/head 2025-12-04T08:57:44.3699477Z * [new branch] gh/oulgen/10/orig -> origin/gh/oulgen/10/orig 2025-12-04T08:57:44.3700955Z * [new branch] gh/oulgen/11/base -> origin/gh/oulgen/11/base 2025-12-04T08:57:44.3702030Z * [new branch] gh/oulgen/11/head -> origin/gh/oulgen/11/head 2025-12-04T08:57:44.3703178Z * [new branch] gh/oulgen/11/orig -> origin/gh/oulgen/11/orig 2025-12-04T08:57:44.3704603Z * [new branch] gh/oulgen/12/base -> origin/gh/oulgen/12/base 2025-12-04T08:57:44.3705766Z * [new branch] gh/oulgen/12/head -> origin/gh/oulgen/12/head 2025-12-04T08:57:44.3706851Z * [new branch] gh/oulgen/12/orig -> origin/gh/oulgen/12/orig 2025-12-04T08:57:44.3708625Z * [new branch] gh/oulgen/13/base -> origin/gh/oulgen/13/base 2025-12-04T08:57:44.3709771Z * [new branch] gh/oulgen/13/head -> origin/gh/oulgen/13/head 2025-12-04T08:57:44.3711024Z * [new branch] gh/oulgen/13/orig -> origin/gh/oulgen/13/orig 2025-12-04T08:57:44.3712494Z * [new branch] gh/oulgen/14/base -> origin/gh/oulgen/14/base 2025-12-04T08:57:44.3713607Z * [new branch] gh/oulgen/14/head -> origin/gh/oulgen/14/head 2025-12-04T08:57:44.3714751Z * [new branch] gh/oulgen/14/orig -> origin/gh/oulgen/14/orig 2025-12-04T08:57:44.3716242Z * [new branch] gh/oulgen/15/base -> origin/gh/oulgen/15/base 2025-12-04T08:57:44.3717345Z * [new branch] gh/oulgen/15/head -> origin/gh/oulgen/15/head 2025-12-04T08:57:44.3718535Z * [new branch] gh/oulgen/15/orig -> origin/gh/oulgen/15/orig 2025-12-04T08:57:44.3720238Z * [new branch] gh/oulgen/16/base -> origin/gh/oulgen/16/base 2025-12-04T08:57:44.3721271Z * [new branch] gh/oulgen/16/head -> origin/gh/oulgen/16/head 2025-12-04T08:57:44.3722339Z * [new branch] gh/oulgen/16/orig -> origin/gh/oulgen/16/orig 2025-12-04T08:57:44.3724083Z * [new branch] gh/oulgen/17/base -> origin/gh/oulgen/17/base 2025-12-04T08:57:44.3725323Z * [new branch] gh/oulgen/17/head -> origin/gh/oulgen/17/head 2025-12-04T08:57:44.3726397Z * [new branch] gh/oulgen/17/orig -> origin/gh/oulgen/17/orig 2025-12-04T08:57:44.3727898Z * [new branch] gh/oulgen/18/base -> origin/gh/oulgen/18/base 2025-12-04T08:57:44.3728978Z * [new branch] gh/oulgen/18/head -> origin/gh/oulgen/18/head 2025-12-04T08:57:44.3730140Z * [new branch] gh/oulgen/18/orig -> origin/gh/oulgen/18/orig 2025-12-04T08:57:44.3731707Z * [new branch] gh/oulgen/19/base -> origin/gh/oulgen/19/base 2025-12-04T08:57:44.3732655Z * [new branch] gh/oulgen/19/head -> origin/gh/oulgen/19/head 2025-12-04T08:57:44.3733838Z * [new branch] gh/oulgen/19/orig -> origin/gh/oulgen/19/orig 2025-12-04T08:57:44.3735834Z * [new branch] gh/oulgen/20/base -> origin/gh/oulgen/20/base 2025-12-04T08:57:44.3737231Z * [new branch] gh/oulgen/20/head -> origin/gh/oulgen/20/head 2025-12-04T08:57:44.3738298Z * [new branch] gh/oulgen/20/orig -> origin/gh/oulgen/20/orig 2025-12-04T08:57:44.3739631Z * [new branch] gh/oulgen/21/base -> origin/gh/oulgen/21/base 2025-12-04T08:57:44.3741076Z * [new branch] gh/oulgen/21/head -> origin/gh/oulgen/21/head 2025-12-04T08:57:44.3742231Z * [new branch] gh/oulgen/21/orig -> origin/gh/oulgen/21/orig 2025-12-04T08:57:44.3743682Z * [new branch] gh/oulgen/22/base -> origin/gh/oulgen/22/base 2025-12-04T08:57:44.3744764Z * [new branch] gh/oulgen/22/head -> origin/gh/oulgen/22/head 2025-12-04T08:57:44.3745893Z * [new branch] gh/oulgen/22/orig -> origin/gh/oulgen/22/orig 2025-12-04T08:57:44.3747360Z * [new branch] gh/oulgen/23/base -> origin/gh/oulgen/23/base 2025-12-04T08:57:44.3748750Z * [new branch] gh/oulgen/23/head -> origin/gh/oulgen/23/head 2025-12-04T08:57:44.3749833Z * [new branch] gh/oulgen/23/orig -> origin/gh/oulgen/23/orig 2025-12-04T08:57:44.3751188Z * [new branch] gh/oulgen/24/base -> origin/gh/oulgen/24/base 2025-12-04T08:57:44.3752393Z * [new branch] gh/oulgen/24/head -> origin/gh/oulgen/24/head 2025-12-04T08:57:44.3753480Z * [new branch] gh/oulgen/24/orig -> origin/gh/oulgen/24/orig 2025-12-04T08:57:44.3754964Z * [new branch] gh/oulgen/25/base -> origin/gh/oulgen/25/base 2025-12-04T08:57:44.3756027Z * [new branch] gh/oulgen/25/head -> origin/gh/oulgen/25/head 2025-12-04T08:57:44.3757155Z * [new branch] gh/oulgen/25/orig -> origin/gh/oulgen/25/orig 2025-12-04T08:57:44.3758604Z * [new branch] gh/oulgen/26/base -> origin/gh/oulgen/26/base 2025-12-04T08:57:44.3759812Z * [new branch] gh/oulgen/26/head -> origin/gh/oulgen/26/head 2025-12-04T08:57:44.3760914Z * [new branch] gh/oulgen/26/orig -> origin/gh/oulgen/26/orig 2025-12-04T08:57:44.3762377Z * [new branch] gh/oulgen/4/base -> origin/gh/oulgen/4/base 2025-12-04T08:57:44.3763484Z * [new branch] gh/oulgen/4/head -> origin/gh/oulgen/4/head 2025-12-04T08:57:44.3764561Z * [new branch] gh/oulgen/4/orig -> origin/gh/oulgen/4/orig 2025-12-04T08:57:44.3766364Z * [new branch] gh/oulgen/7/base -> origin/gh/oulgen/7/base 2025-12-04T08:57:44.3767579Z * [new branch] gh/oulgen/7/head -> origin/gh/oulgen/7/head 2025-12-04T08:57:44.3768649Z * [new branch] gh/oulgen/7/orig -> origin/gh/oulgen/7/orig 2025-12-04T08:57:44.3770169Z * [new branch] gh/oulgen/8/base -> origin/gh/oulgen/8/base 2025-12-04T08:57:44.3771197Z * [new branch] gh/oulgen/8/head -> origin/gh/oulgen/8/head 2025-12-04T08:57:44.3772290Z * [new branch] gh/oulgen/8/orig -> origin/gh/oulgen/8/orig 2025-12-04T08:57:44.3773757Z * [new branch] gh/oulgen/9/base -> origin/gh/oulgen/9/base 2025-12-04T08:57:44.3774843Z * [new branch] gh/oulgen/9/head -> origin/gh/oulgen/9/head 2025-12-04T08:57:44.3776078Z * [new branch] gh/oulgen/9/orig -> origin/gh/oulgen/9/orig 2025-12-04T08:57:44.3777736Z * [new branch] gh/patvig/mtia-serialization -> origin/gh/patvig/mtia-serialization 2025-12-04T08:57:44.3779526Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-12-04T08:57:44.3780664Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-12-04T08:57:44.3781905Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-12-04T08:57:44.3783335Z * [new branch] gh/pearu/109/base -> origin/gh/pearu/109/base 2025-12-04T08:57:44.3784447Z * [new branch] gh/pearu/109/head -> origin/gh/pearu/109/head 2025-12-04T08:57:44.3785584Z * [new branch] gh/pearu/109/orig -> origin/gh/pearu/109/orig 2025-12-04T08:57:44.3787091Z * [new branch] gh/pearu/110/base -> origin/gh/pearu/110/base 2025-12-04T08:57:44.3788597Z * [new branch] gh/pearu/110/head -> origin/gh/pearu/110/head 2025-12-04T08:57:44.3789744Z * [new branch] gh/pearu/110/orig -> origin/gh/pearu/110/orig 2025-12-04T08:57:44.3791292Z * [new branch] gh/pearu/111/base -> origin/gh/pearu/111/base 2025-12-04T08:57:44.3792382Z * [new branch] gh/pearu/111/head -> origin/gh/pearu/111/head 2025-12-04T08:57:44.3793543Z * [new branch] gh/pearu/111/orig -> origin/gh/pearu/111/orig 2025-12-04T08:57:44.3795066Z * [new branch] gh/pearu/112/base -> origin/gh/pearu/112/base 2025-12-04T08:57:44.3796211Z * [new branch] gh/pearu/112/head -> origin/gh/pearu/112/head 2025-12-04T08:57:44.3797460Z * [new branch] gh/pearu/112/orig -> origin/gh/pearu/112/orig 2025-12-04T08:57:44.3798859Z * [new branch] gh/pearu/115/base -> origin/gh/pearu/115/base 2025-12-04T08:57:44.3800123Z * [new branch] gh/pearu/115/head -> origin/gh/pearu/115/head 2025-12-04T08:57:44.3801229Z * [new branch] gh/pearu/115/orig -> origin/gh/pearu/115/orig 2025-12-04T08:57:44.3802572Z * [new branch] gh/pearu/116/base -> origin/gh/pearu/116/base 2025-12-04T08:57:44.3803649Z * [new branch] gh/pearu/116/head -> origin/gh/pearu/116/head 2025-12-04T08:57:44.3804721Z * [new branch] gh/pearu/116/orig -> origin/gh/pearu/116/orig 2025-12-04T08:57:44.3806143Z * [new branch] gh/pearu/117/base -> origin/gh/pearu/117/base 2025-12-04T08:57:44.3807250Z * [new branch] gh/pearu/117/head -> origin/gh/pearu/117/head 2025-12-04T08:57:44.3808363Z * [new branch] gh/pearu/117/orig -> origin/gh/pearu/117/orig 2025-12-04T08:57:44.3809888Z * [new branch] gh/pearu/118/base -> origin/gh/pearu/118/base 2025-12-04T08:57:44.3810946Z * [new branch] gh/pearu/118/head -> origin/gh/pearu/118/head 2025-12-04T08:57:44.3812175Z * [new branch] gh/pearu/118/orig -> origin/gh/pearu/118/orig 2025-12-04T08:57:44.3813543Z * [new branch] gh/pearu/119/base -> origin/gh/pearu/119/base 2025-12-04T08:57:44.3814689Z * [new branch] gh/pearu/119/head -> origin/gh/pearu/119/head 2025-12-04T08:57:44.3815774Z * [new branch] gh/pearu/119/orig -> origin/gh/pearu/119/orig 2025-12-04T08:57:44.3817202Z * [new branch] gh/pearu/139/base -> origin/gh/pearu/139/base 2025-12-04T08:57:44.3818291Z * [new branch] gh/pearu/139/head -> origin/gh/pearu/139/head 2025-12-04T08:57:44.3819416Z * [new branch] gh/pearu/139/orig -> origin/gh/pearu/139/orig 2025-12-04T08:57:44.3820870Z * [new branch] gh/pearu/140/base -> origin/gh/pearu/140/base 2025-12-04T08:57:44.3821923Z * [new branch] gh/pearu/140/head -> origin/gh/pearu/140/head 2025-12-04T08:57:44.3823131Z * [new branch] gh/pearu/140/orig -> origin/gh/pearu/140/orig 2025-12-04T08:57:44.3825030Z * [new branch] gh/pearu/142/base -> origin/gh/pearu/142/base 2025-12-04T08:57:44.3826153Z * [new branch] gh/pearu/142/head -> origin/gh/pearu/142/head 2025-12-04T08:57:44.3827460Z * [new branch] gh/pearu/142/orig -> origin/gh/pearu/142/orig 2025-12-04T08:57:44.3828992Z * [new branch] gh/pearu/143/base -> origin/gh/pearu/143/base 2025-12-04T08:57:44.3830157Z * [new branch] gh/pearu/143/head -> origin/gh/pearu/143/head 2025-12-04T08:57:44.3831256Z * [new branch] gh/pearu/143/orig -> origin/gh/pearu/143/orig 2025-12-04T08:57:44.3832850Z * [new branch] gh/pearu/147/base -> origin/gh/pearu/147/base 2025-12-04T08:57:44.3833976Z * [new branch] gh/pearu/147/head -> origin/gh/pearu/147/head 2025-12-04T08:57:44.3835117Z * [new branch] gh/pearu/147/orig -> origin/gh/pearu/147/orig 2025-12-04T08:57:44.3836630Z * [new branch] gh/pearu/149/base -> origin/gh/pearu/149/base 2025-12-04T08:57:44.3837728Z * [new branch] gh/pearu/149/head -> origin/gh/pearu/149/head 2025-12-04T08:57:44.3839414Z * [new branch] gh/pearu/149/orig -> origin/gh/pearu/149/orig 2025-12-04T08:57:44.3841215Z * [new branch] gh/pearu/150/base -> origin/gh/pearu/150/base 2025-12-04T08:57:44.3842468Z * [new branch] gh/pearu/150/head -> origin/gh/pearu/150/head 2025-12-04T08:57:44.3843717Z * [new branch] gh/pearu/150/orig -> origin/gh/pearu/150/orig 2025-12-04T08:57:44.3845250Z * [new branch] gh/pearu/151/base -> origin/gh/pearu/151/base 2025-12-04T08:57:44.3846412Z * [new branch] gh/pearu/151/head -> origin/gh/pearu/151/head 2025-12-04T08:57:44.3847485Z * [new branch] gh/pearu/151/orig -> origin/gh/pearu/151/orig 2025-12-04T08:57:44.3848981Z * [new branch] gh/pearu/152/base -> origin/gh/pearu/152/base 2025-12-04T08:57:44.3850125Z * [new branch] gh/pearu/152/head -> origin/gh/pearu/152/head 2025-12-04T08:57:44.3851212Z * [new branch] gh/pearu/152/orig -> origin/gh/pearu/152/orig 2025-12-04T08:57:44.3852813Z * [new branch] gh/pearu/153/base -> origin/gh/pearu/153/base 2025-12-04T08:57:44.3853759Z * [new branch] gh/pearu/153/head -> origin/gh/pearu/153/head 2025-12-04T08:57:44.3854862Z * [new branch] gh/pearu/153/orig -> origin/gh/pearu/153/orig 2025-12-04T08:57:44.3856313Z * [new branch] gh/pearu/154/base -> origin/gh/pearu/154/base 2025-12-04T08:57:44.3857367Z * [new branch] gh/pearu/154/head -> origin/gh/pearu/154/head 2025-12-04T08:57:44.3858537Z * [new branch] gh/pearu/154/orig -> origin/gh/pearu/154/orig 2025-12-04T08:57:44.3860107Z * [new branch] gh/pearu/155/base -> origin/gh/pearu/155/base 2025-12-04T08:57:44.3861328Z * [new branch] gh/pearu/155/head -> origin/gh/pearu/155/head 2025-12-04T08:57:44.3862327Z * [new branch] gh/pearu/155/orig -> origin/gh/pearu/155/orig 2025-12-04T08:57:44.3863852Z * [new branch] gh/pearu/156/base -> origin/gh/pearu/156/base 2025-12-04T08:57:44.3864914Z * [new branch] gh/pearu/156/head -> origin/gh/pearu/156/head 2025-12-04T08:57:44.3866006Z * [new branch] gh/pearu/156/orig -> origin/gh/pearu/156/orig 2025-12-04T08:57:44.3868260Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-12-04T08:57:44.3869599Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-12-04T08:57:44.3870933Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-12-04T08:57:44.3872538Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-12-04T08:57:44.3873703Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-12-04T08:57:44.3874983Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-12-04T08:57:44.3876793Z * [new branch] gh/pianpwk/21/base -> origin/gh/pianpwk/21/base 2025-12-04T08:57:44.3877953Z * [new branch] gh/pianpwk/21/head -> origin/gh/pianpwk/21/head 2025-12-04T08:57:44.3879502Z * [new branch] gh/pianpwk/28/base -> origin/gh/pianpwk/28/base 2025-12-04T08:57:44.3880707Z * [new branch] gh/pianpwk/28/head -> origin/gh/pianpwk/28/head 2025-12-04T08:57:44.3881802Z * [new branch] gh/pianpwk/28/orig -> origin/gh/pianpwk/28/orig 2025-12-04T08:57:44.3883376Z * [new branch] gh/pianpwk/29/base -> origin/gh/pianpwk/29/base 2025-12-04T08:57:44.3884562Z * [new branch] gh/pianpwk/29/head -> origin/gh/pianpwk/29/head 2025-12-04T08:57:44.3885651Z * [new branch] gh/pianpwk/29/orig -> origin/gh/pianpwk/29/orig 2025-12-04T08:57:44.3887363Z * [new branch] gh/pianpwk/30/base -> origin/gh/pianpwk/30/base 2025-12-04T08:57:44.3888393Z * [new branch] gh/pianpwk/30/head -> origin/gh/pianpwk/30/head 2025-12-04T08:57:44.3889661Z * [new branch] gh/pianpwk/30/orig -> origin/gh/pianpwk/30/orig 2025-12-04T08:57:44.3891148Z * [new branch] gh/pianpwk/31/base -> origin/gh/pianpwk/31/base 2025-12-04T08:57:44.3892254Z * [new branch] gh/pianpwk/31/head -> origin/gh/pianpwk/31/head 2025-12-04T08:57:44.3893356Z * [new branch] gh/pianpwk/31/orig -> origin/gh/pianpwk/31/orig 2025-12-04T08:57:44.3894659Z * [new branch] gh/pianpwk/32/base -> origin/gh/pianpwk/32/base 2025-12-04T08:57:44.3895736Z * [new branch] gh/pianpwk/32/head -> origin/gh/pianpwk/32/head 2025-12-04T08:57:44.3896812Z * [new branch] gh/pianpwk/32/orig -> origin/gh/pianpwk/32/orig 2025-12-04T08:57:44.3898142Z * [new branch] gh/pianpwk/33/base -> origin/gh/pianpwk/33/base 2025-12-04T08:57:44.3899202Z * [new branch] gh/pianpwk/33/head -> origin/gh/pianpwk/33/head 2025-12-04T08:57:44.3900275Z * [new branch] gh/pianpwk/33/orig -> origin/gh/pianpwk/33/orig 2025-12-04T08:57:44.3902067Z * [new branch] gh/pianpwk/34/base -> origin/gh/pianpwk/34/base 2025-12-04T08:57:44.3903508Z * [new branch] gh/pianpwk/34/head -> origin/gh/pianpwk/34/head 2025-12-04T08:57:44.3904942Z * [new branch] gh/pianpwk/34/orig -> origin/gh/pianpwk/34/orig 2025-12-04T08:57:44.3906377Z * [new branch] gh/pianpwk/35/base -> origin/gh/pianpwk/35/base 2025-12-04T08:57:44.3907869Z * [new branch] gh/pianpwk/35/head -> origin/gh/pianpwk/35/head 2025-12-04T08:57:44.3909403Z * [new branch] gh/pianpwk/35/orig -> origin/gh/pianpwk/35/orig 2025-12-04T08:57:44.3911165Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-12-04T08:57:44.3912332Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-12-04T08:57:44.3913825Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-12-04T08:57:44.3914938Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-12-04T08:57:44.3916049Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-12-04T08:57:44.3917501Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-12-04T08:57:44.3918745Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-12-04T08:57:44.3920008Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-12-04T08:57:44.3921517Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-12-04T08:57:44.3922571Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-12-04T08:57:44.3923844Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-12-04T08:57:44.3927772Z * [new branch] gh/rec/166/base -> origin/gh/rec/166/base 2025-12-04T08:57:44.3928912Z * [new branch] gh/rec/166/head -> origin/gh/rec/166/head 2025-12-04T08:57:44.3930026Z * [new branch] gh/rec/166/orig -> origin/gh/rec/166/orig 2025-12-04T08:57:44.3932014Z * [new branch] gh/rec/167/base -> origin/gh/rec/167/base 2025-12-04T08:57:44.3933173Z * [new branch] gh/rec/167/head -> origin/gh/rec/167/head 2025-12-04T08:57:44.3934295Z * [new branch] gh/rec/167/orig -> origin/gh/rec/167/orig 2025-12-04T08:57:44.3935848Z * [new branch] gh/rec/168/base -> origin/gh/rec/168/base 2025-12-04T08:57:44.3937090Z * [new branch] gh/rec/168/head -> origin/gh/rec/168/head 2025-12-04T08:57:44.3938714Z * [new branch] gh/rec/168/orig -> origin/gh/rec/168/orig 2025-12-04T08:57:44.3940174Z * [new branch] gh/rec/169/base -> origin/gh/rec/169/base 2025-12-04T08:57:44.3941261Z * [new branch] gh/rec/169/head -> origin/gh/rec/169/head 2025-12-04T08:57:44.3942354Z * [new branch] gh/rec/169/orig -> origin/gh/rec/169/orig 2025-12-04T08:57:44.3943765Z * [new branch] gh/rec/170/base -> origin/gh/rec/170/base 2025-12-04T08:57:44.3944858Z * [new branch] gh/rec/170/head -> origin/gh/rec/170/head 2025-12-04T08:57:44.3945955Z * [new branch] gh/rec/170/orig -> origin/gh/rec/170/orig 2025-12-04T08:57:44.3947425Z * [new branch] gh/rec/171/base -> origin/gh/rec/171/base 2025-12-04T08:57:44.3948833Z * [new branch] gh/rec/171/head -> origin/gh/rec/171/head 2025-12-04T08:57:44.3950003Z * [new branch] gh/rec/171/orig -> origin/gh/rec/171/orig 2025-12-04T08:57:44.3951483Z * [new branch] gh/rec/172/base -> origin/gh/rec/172/base 2025-12-04T08:57:44.3952560Z * [new branch] gh/rec/172/head -> origin/gh/rec/172/head 2025-12-04T08:57:44.3953748Z * [new branch] gh/rec/172/orig -> origin/gh/rec/172/orig 2025-12-04T08:57:44.3955251Z * [new branch] gh/rec/173/base -> origin/gh/rec/173/base 2025-12-04T08:57:44.3956382Z * [new branch] gh/rec/173/head -> origin/gh/rec/173/head 2025-12-04T08:57:44.3957560Z * [new branch] gh/rec/173/orig -> origin/gh/rec/173/orig 2025-12-04T08:57:44.3959122Z * [new branch] gh/rec/174/base -> origin/gh/rec/174/base 2025-12-04T08:57:44.3960336Z * [new branch] gh/rec/174/head -> origin/gh/rec/174/head 2025-12-04T08:57:44.3961440Z * [new branch] gh/rec/174/orig -> origin/gh/rec/174/orig 2025-12-04T08:57:44.3962839Z * [new branch] gh/rec/175/base -> origin/gh/rec/175/base 2025-12-04T08:57:44.3963905Z * [new branch] gh/rec/175/head -> origin/gh/rec/175/head 2025-12-04T08:57:44.3964991Z * [new branch] gh/rec/175/orig -> origin/gh/rec/175/orig 2025-12-04T08:57:44.3966419Z * [new branch] gh/rec/176/base -> origin/gh/rec/176/base 2025-12-04T08:57:44.3967462Z * [new branch] gh/rec/176/head -> origin/gh/rec/176/head 2025-12-04T08:57:44.3968790Z * [new branch] gh/rec/176/orig -> origin/gh/rec/176/orig 2025-12-04T08:57:44.3970089Z * [new branch] gh/rec/177/base -> origin/gh/rec/177/base 2025-12-04T08:57:44.3971235Z * [new branch] gh/rec/177/head -> origin/gh/rec/177/head 2025-12-04T08:57:44.3972296Z * [new branch] gh/rec/177/orig -> origin/gh/rec/177/orig 2025-12-04T08:57:44.3974084Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-12-04T08:57:44.3975204Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-12-04T08:57:44.3976307Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-12-04T08:57:44.3977858Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-12-04T08:57:44.3978993Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-12-04T08:57:44.3980084Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-12-04T08:57:44.3981539Z * [new branch] gh/robert-hardwick/5/base -> origin/gh/robert-hardwick/5/base 2025-12-04T08:57:44.3982741Z * [new branch] gh/robert-hardwick/5/head -> origin/gh/robert-hardwick/5/head 2025-12-04T08:57:44.3983903Z * [new branch] gh/robert-hardwick/5/orig -> origin/gh/robert-hardwick/5/orig 2025-12-04T08:57:44.3985382Z * [new branch] gh/robert-hardwick/6/base -> origin/gh/robert-hardwick/6/base 2025-12-04T08:57:44.3986472Z * [new branch] gh/robert-hardwick/6/head -> origin/gh/robert-hardwick/6/head 2025-12-04T08:57:44.3987790Z * [new branch] gh/robert-hardwick/6/orig -> origin/gh/robert-hardwick/6/orig 2025-12-04T08:57:44.3989369Z * [new branch] gh/robert-hardwick/7/base -> origin/gh/robert-hardwick/7/base 2025-12-04T08:57:44.3990500Z * [new branch] gh/robert-hardwick/7/head -> origin/gh/robert-hardwick/7/head 2025-12-04T08:57:44.3991630Z * [new branch] gh/robert-hardwick/7/orig -> origin/gh/robert-hardwick/7/orig 2025-12-04T08:57:44.3993124Z * [new branch] gh/robert-hardwick/8/base -> origin/gh/robert-hardwick/8/base 2025-12-04T08:57:44.3994258Z * [new branch] gh/robert-hardwick/8/head -> origin/gh/robert-hardwick/8/head 2025-12-04T08:57:44.3995466Z * [new branch] gh/robert-hardwick/8/orig -> origin/gh/robert-hardwick/8/orig 2025-12-04T08:57:44.3996960Z * [new branch] gh/robert-hardwick/9/base -> origin/gh/robert-hardwick/9/base 2025-12-04T08:57:44.3998667Z * [new branch] gh/robert-hardwick/9/head -> origin/gh/robert-hardwick/9/head 2025-12-04T08:57:44.3999913Z * [new branch] gh/robert-hardwick/9/orig -> origin/gh/robert-hardwick/9/orig 2025-12-04T08:57:44.4001618Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-12-04T08:57:44.4002750Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-12-04T08:57:44.4004552Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-12-04T08:57:44.4005691Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-12-04T08:57:44.4007174Z * [new branch] gh/rtimpe/22/base -> origin/gh/rtimpe/22/base 2025-12-04T08:57:44.4008709Z * [new branch] gh/rtimpe/22/head -> origin/gh/rtimpe/22/head 2025-12-04T08:57:44.4009829Z * [new branch] gh/rtimpe/22/orig -> origin/gh/rtimpe/22/orig 2025-12-04T08:57:44.4011199Z * [new branch] gh/rtimpe/23/base -> origin/gh/rtimpe/23/base 2025-12-04T08:57:44.4012332Z * [new branch] gh/rtimpe/23/head -> origin/gh/rtimpe/23/head 2025-12-04T08:57:44.4013569Z * [new branch] gh/rtimpe/23/orig -> origin/gh/rtimpe/23/orig 2025-12-04T08:57:44.4014948Z * [new branch] gh/rtimpe/24/base -> origin/gh/rtimpe/24/base 2025-12-04T08:57:44.4015997Z * [new branch] gh/rtimpe/24/head -> origin/gh/rtimpe/24/head 2025-12-04T08:57:44.4017085Z * [new branch] gh/rtimpe/24/orig -> origin/gh/rtimpe/24/orig 2025-12-04T08:57:44.4018525Z * [new branch] gh/rtimpe/25/base -> origin/gh/rtimpe/25/base 2025-12-04T08:57:44.4019607Z * [new branch] gh/rtimpe/25/head -> origin/gh/rtimpe/25/head 2025-12-04T08:57:44.4020787Z * [new branch] gh/rtimpe/25/orig -> origin/gh/rtimpe/25/orig 2025-12-04T08:57:44.4022228Z * [new branch] gh/rtimpe/26/base -> origin/gh/rtimpe/26/base 2025-12-04T08:57:44.4023300Z * [new branch] gh/rtimpe/26/head -> origin/gh/rtimpe/26/head 2025-12-04T08:57:44.4024651Z * [new branch] gh/rtimpe/26/orig -> origin/gh/rtimpe/26/orig 2025-12-04T08:57:44.4026218Z * [new branch] gh/rtimpe/27/base -> origin/gh/rtimpe/27/base 2025-12-04T08:57:44.4027487Z * [new branch] gh/rtimpe/27/head -> origin/gh/rtimpe/27/head 2025-12-04T08:57:44.4028740Z * [new branch] gh/rtimpe/27/orig -> origin/gh/rtimpe/27/orig 2025-12-04T08:57:44.4030227Z * [new branch] gh/rtimpe/28/base -> origin/gh/rtimpe/28/base 2025-12-04T08:57:44.4031323Z * [new branch] gh/rtimpe/28/head -> origin/gh/rtimpe/28/head 2025-12-04T08:57:44.4032500Z * [new branch] gh/rtimpe/28/orig -> origin/gh/rtimpe/28/orig 2025-12-04T08:57:44.4034054Z * [new branch] gh/rtimpe/29/base -> origin/gh/rtimpe/29/base 2025-12-04T08:57:44.4035145Z * [new branch] gh/rtimpe/29/head -> origin/gh/rtimpe/29/head 2025-12-04T08:57:44.4036302Z * [new branch] gh/rtimpe/29/orig -> origin/gh/rtimpe/29/orig 2025-12-04T08:57:44.4037710Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-12-04T08:57:44.4038756Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-12-04T08:57:44.4040269Z * [new branch] gh/rtimpe/30/base -> origin/gh/rtimpe/30/base 2025-12-04T08:57:44.4041323Z * [new branch] gh/rtimpe/30/head -> origin/gh/rtimpe/30/head 2025-12-04T08:57:44.4042510Z * [new branch] gh/rtimpe/30/orig -> origin/gh/rtimpe/30/orig 2025-12-04T08:57:44.4044021Z * [new branch] gh/rtimpe/31/base -> origin/gh/rtimpe/31/base 2025-12-04T08:57:44.4045121Z * [new branch] gh/rtimpe/31/head -> origin/gh/rtimpe/31/head 2025-12-04T08:57:44.4046292Z * [new branch] gh/rtimpe/31/orig -> origin/gh/rtimpe/31/orig 2025-12-04T08:57:44.4047816Z * [new branch] gh/rtimpe/32/base -> origin/gh/rtimpe/32/base 2025-12-04T08:57:44.4048893Z * [new branch] gh/rtimpe/32/head -> origin/gh/rtimpe/32/head 2025-12-04T08:57:44.4049968Z * [new branch] gh/rtimpe/32/orig -> origin/gh/rtimpe/32/orig 2025-12-04T08:57:44.4051405Z * [new branch] gh/rtimpe/33/base -> origin/gh/rtimpe/33/base 2025-12-04T08:57:44.4052462Z * [new branch] gh/rtimpe/33/head -> origin/gh/rtimpe/33/head 2025-12-04T08:57:44.4053587Z * [new branch] gh/rtimpe/33/orig -> origin/gh/rtimpe/33/orig 2025-12-04T08:57:44.4054895Z * [new branch] gh/rtimpe/34/base -> origin/gh/rtimpe/34/base 2025-12-04T08:57:44.4055981Z * [new branch] gh/rtimpe/34/head -> origin/gh/rtimpe/34/head 2025-12-04T08:57:44.4057082Z * [new branch] gh/rtimpe/34/orig -> origin/gh/rtimpe/34/orig 2025-12-04T08:57:44.4058781Z * [new branch] gh/rtimpe/35/base -> origin/gh/rtimpe/35/base 2025-12-04T08:57:44.4059783Z * [new branch] gh/rtimpe/35/head -> origin/gh/rtimpe/35/head 2025-12-04T08:57:44.4060890Z * [new branch] gh/rtimpe/35/orig -> origin/gh/rtimpe/35/orig 2025-12-04T08:57:44.4062321Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-12-04T08:57:44.4063428Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-12-04T08:57:44.4065313Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-12-04T08:57:44.4066394Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-12-04T08:57:44.4067585Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-12-04T08:57:44.4069374Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-12-04T08:57:44.4070560Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-12-04T08:57:44.4071685Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-12-04T08:57:44.4073252Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-12-04T08:57:44.4074380Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-12-04T08:57:44.4075473Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-12-04T08:57:44.4077076Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-12-04T08:57:44.4078206Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-12-04T08:57:44.4079304Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-12-04T08:57:44.4080853Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-12-04T08:57:44.4082005Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-12-04T08:57:44.4083086Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-12-04T08:57:44.4084460Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-12-04T08:57:44.4085542Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-12-04T08:57:44.4086669Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-12-04T08:57:44.4088199Z * [new branch] gh/ruisizhang123/9/base -> origin/gh/ruisizhang123/9/base 2025-12-04T08:57:44.4089310Z * [new branch] gh/ruisizhang123/9/head -> origin/gh/ruisizhang123/9/head 2025-12-04T08:57:44.4090410Z * [new branch] gh/ruisizhang123/9/orig -> origin/gh/ruisizhang123/9/orig 2025-12-04T08:57:44.4092255Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-12-04T08:57:44.4093296Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-12-04T08:57:44.4094424Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-12-04T08:57:44.4095866Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-12-04T08:57:44.4097005Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-12-04T08:57:44.4098029Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-12-04T08:57:44.4099475Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-12-04T08:57:44.4100572Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-12-04T08:57:44.4101827Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-12-04T08:57:44.4103231Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-12-04T08:57:44.4104173Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-12-04T08:57:44.4105366Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-12-04T08:57:44.4106903Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-12-04T08:57:44.4108324Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-12-04T08:57:44.4109471Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-12-04T08:57:44.4110935Z * [new branch] gh/seemethere/62/base -> origin/gh/seemethere/62/base 2025-12-04T08:57:44.4112078Z * [new branch] gh/seemethere/62/head -> origin/gh/seemethere/62/head 2025-12-04T08:57:44.4113217Z * [new branch] gh/seemethere/62/orig -> origin/gh/seemethere/62/orig 2025-12-04T08:57:44.4114665Z * [new branch] gh/seemethere/63/base -> origin/gh/seemethere/63/base 2025-12-04T08:57:44.4115840Z * [new branch] gh/seemethere/63/head -> origin/gh/seemethere/63/head 2025-12-04T08:57:44.4117081Z * [new branch] gh/seemethere/63/orig -> origin/gh/seemethere/63/orig 2025-12-04T08:57:44.4118556Z * [new branch] gh/seemethere/71/base -> origin/gh/seemethere/71/base 2025-12-04T08:57:44.4119781Z * [new branch] gh/seemethere/71/head -> origin/gh/seemethere/71/head 2025-12-04T08:57:44.4120884Z * [new branch] gh/seemethere/71/orig -> origin/gh/seemethere/71/orig 2025-12-04T08:57:44.4122388Z * [new branch] gh/seemethere/72/base -> origin/gh/seemethere/72/base 2025-12-04T08:57:44.4123449Z * [new branch] gh/seemethere/72/head -> origin/gh/seemethere/72/head 2025-12-04T08:57:44.4125065Z * [new branch] gh/seemethere/72/orig -> origin/gh/seemethere/72/orig 2025-12-04T08:57:44.4126476Z * [new branch] gh/seemethere/73/base -> origin/gh/seemethere/73/base 2025-12-04T08:57:44.4127590Z * [new branch] gh/seemethere/73/head -> origin/gh/seemethere/73/head 2025-12-04T08:57:44.4128783Z * [new branch] gh/seemethere/73/orig -> origin/gh/seemethere/73/orig 2025-12-04T08:57:44.4130296Z * [new branch] gh/seemethere/74/base -> origin/gh/seemethere/74/base 2025-12-04T08:57:44.4131407Z * [new branch] gh/seemethere/74/head -> origin/gh/seemethere/74/head 2025-12-04T08:57:44.4133080Z * [new branch] gh/seemethere/74/orig -> origin/gh/seemethere/74/orig 2025-12-04T08:57:44.4134602Z * [new branch] gh/seemethere/75/base -> origin/gh/seemethere/75/base 2025-12-04T08:57:44.4135767Z * [new branch] gh/seemethere/75/head -> origin/gh/seemethere/75/head 2025-12-04T08:57:44.4137031Z * [new branch] gh/seemethere/75/orig -> origin/gh/seemethere/75/orig 2025-12-04T08:57:44.4138436Z * [new branch] gh/seemethere/76/base -> origin/gh/seemethere/76/base 2025-12-04T08:57:44.4139506Z * [new branch] gh/seemethere/76/head -> origin/gh/seemethere/76/head 2025-12-04T08:57:44.4140615Z * [new branch] gh/seemethere/76/orig -> origin/gh/seemethere/76/orig 2025-12-04T08:57:44.4142610Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-12-04T08:57:44.4143848Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-12-04T08:57:44.4145064Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-12-04T08:57:44.4146767Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-12-04T08:57:44.4148729Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-12-04T08:57:44.4149762Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-12-04T08:57:44.4151386Z * [new branch] gh/shunting314/249/base -> origin/gh/shunting314/249/base 2025-12-04T08:57:44.4152611Z * [new branch] gh/shunting314/249/head -> origin/gh/shunting314/249/head 2025-12-04T08:57:44.4153893Z * [new branch] gh/shunting314/249/orig -> origin/gh/shunting314/249/orig 2025-12-04T08:57:44.4155436Z * [new branch] gh/shunting314/253/base -> origin/gh/shunting314/253/base 2025-12-04T08:57:44.4156520Z * [new branch] gh/shunting314/253/head -> origin/gh/shunting314/253/head 2025-12-04T08:57:44.4157652Z * [new branch] gh/shunting314/253/orig -> origin/gh/shunting314/253/orig 2025-12-04T08:57:44.4159197Z * [new branch] gh/shunting314/256/base -> origin/gh/shunting314/256/base 2025-12-04T08:57:44.4160422Z * [new branch] gh/shunting314/256/head -> origin/gh/shunting314/256/head 2025-12-04T08:57:44.4161495Z * [new branch] gh/shunting314/256/orig -> origin/gh/shunting314/256/orig 2025-12-04T08:57:44.4163326Z * [new branch] gh/shunting314/257/base -> origin/gh/shunting314/257/base 2025-12-04T08:57:44.4165071Z * [new branch] gh/shunting314/257/head -> origin/gh/shunting314/257/head 2025-12-04T08:57:44.4166164Z * [new branch] gh/shunting314/257/orig -> origin/gh/shunting314/257/orig 2025-12-04T08:57:44.4167829Z * [new branch] gh/shunting314/258/base -> origin/gh/shunting314/258/base 2025-12-04T08:57:44.4168852Z * [new branch] gh/shunting314/258/head -> origin/gh/shunting314/258/head 2025-12-04T08:57:44.4170036Z * [new branch] gh/shunting314/258/orig -> origin/gh/shunting314/258/orig 2025-12-04T08:57:44.4171322Z * [new branch] gh/shunting314/259/base -> origin/gh/shunting314/259/base 2025-12-04T08:57:44.4172435Z * [new branch] gh/shunting314/259/head -> origin/gh/shunting314/259/head 2025-12-04T08:57:44.4173542Z * [new branch] gh/shunting314/259/orig -> origin/gh/shunting314/259/orig 2025-12-04T08:57:44.4175229Z * [new branch] gh/shunting314/260/base -> origin/gh/shunting314/260/base 2025-12-04T08:57:44.4176394Z * [new branch] gh/shunting314/260/head -> origin/gh/shunting314/260/head 2025-12-04T08:57:44.4177569Z * [new branch] gh/shunting314/260/orig -> origin/gh/shunting314/260/orig 2025-12-04T08:57:44.4179130Z * [new branch] gh/shunting314/261/base -> origin/gh/shunting314/261/base 2025-12-04T08:57:44.4180510Z * [new branch] gh/shunting314/261/head -> origin/gh/shunting314/261/head 2025-12-04T08:57:44.4181637Z * [new branch] gh/shunting314/261/orig -> origin/gh/shunting314/261/orig 2025-12-04T08:57:44.4183217Z * [new branch] gh/shunting314/262/base -> origin/gh/shunting314/262/base 2025-12-04T08:57:44.4184383Z * [new branch] gh/shunting314/262/head -> origin/gh/shunting314/262/head 2025-12-04T08:57:44.4185540Z * [new branch] gh/shunting314/262/orig -> origin/gh/shunting314/262/orig 2025-12-04T08:57:44.4187041Z * [new branch] gh/shunting314/263/base -> origin/gh/shunting314/263/base 2025-12-04T08:57:44.4188648Z * [new branch] gh/shunting314/263/head -> origin/gh/shunting314/263/head 2025-12-04T08:57:44.4189764Z * [new branch] gh/shunting314/263/orig -> origin/gh/shunting314/263/orig 2025-12-04T08:57:44.4191752Z * [new branch] gh/shunting314/264/base -> origin/gh/shunting314/264/base 2025-12-04T08:57:44.4192917Z * [new branch] gh/shunting314/264/head -> origin/gh/shunting314/264/head 2025-12-04T08:57:44.4194176Z * [new branch] gh/shunting314/264/orig -> origin/gh/shunting314/264/orig 2025-12-04T08:57:44.4195739Z * [new branch] gh/shunting314/265/base -> origin/gh/shunting314/265/base 2025-12-04T08:57:44.4196906Z * [new branch] gh/shunting314/265/head -> origin/gh/shunting314/265/head 2025-12-04T08:57:44.4198052Z * [new branch] gh/shunting314/265/orig -> origin/gh/shunting314/265/orig 2025-12-04T08:57:44.4199663Z * [new branch] gh/shunting314/266/base -> origin/gh/shunting314/266/base 2025-12-04T08:57:44.4200988Z * [new branch] gh/shunting314/266/head -> origin/gh/shunting314/266/head 2025-12-04T08:57:44.4202077Z * [new branch] gh/shunting314/266/orig -> origin/gh/shunting314/266/orig 2025-12-04T08:57:44.4203753Z * [new branch] gh/shunting314/267/base -> origin/gh/shunting314/267/base 2025-12-04T08:57:44.4205124Z * [new branch] gh/shunting314/267/head -> origin/gh/shunting314/267/head 2025-12-04T08:57:44.4206274Z * [new branch] gh/shunting314/267/orig -> origin/gh/shunting314/267/orig 2025-12-04T08:57:44.4208542Z * [new branch] gh/shunting314/268/base -> origin/gh/shunting314/268/base 2025-12-04T08:57:44.4209745Z * [new branch] gh/shunting314/268/head -> origin/gh/shunting314/268/head 2025-12-04T08:57:44.4210821Z * [new branch] gh/shunting314/268/orig -> origin/gh/shunting314/268/orig 2025-12-04T08:57:44.4212409Z * [new branch] gh/shunting314/269/base -> origin/gh/shunting314/269/base 2025-12-04T08:57:44.4213606Z * [new branch] gh/shunting314/269/head -> origin/gh/shunting314/269/head 2025-12-04T08:57:44.4214713Z * [new branch] gh/shunting314/269/orig -> origin/gh/shunting314/269/orig 2025-12-04T08:57:44.4216458Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-12-04T08:57:44.4217658Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-12-04T08:57:44.4218918Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-12-04T08:57:44.4219946Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-12-04T08:57:44.4221237Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-12-04T08:57:44.4222367Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-12-04T08:57:44.4223819Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-12-04T08:57:44.4228481Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-12-04T08:57:44.4230864Z * [new branch] gh/slayton58/39/base -> origin/gh/slayton58/39/base 2025-12-04T08:57:44.4232073Z * [new branch] gh/slayton58/39/head -> origin/gh/slayton58/39/head 2025-12-04T08:57:44.4233288Z * [new branch] gh/slayton58/39/orig -> origin/gh/slayton58/39/orig 2025-12-04T08:57:44.4234798Z * [new branch] gh/slayton58/42/base -> origin/gh/slayton58/42/base 2025-12-04T08:57:44.4235934Z * [new branch] gh/slayton58/42/head -> origin/gh/slayton58/42/head 2025-12-04T08:57:44.4237234Z * [new branch] gh/slayton58/42/orig -> origin/gh/slayton58/42/orig 2025-12-04T08:57:44.4238711Z * [new branch] gh/slayton58/43/base -> origin/gh/slayton58/43/base 2025-12-04T08:57:44.4239977Z * [new branch] gh/slayton58/43/head -> origin/gh/slayton58/43/head 2025-12-04T08:57:44.4241098Z * [new branch] gh/slayton58/43/orig -> origin/gh/slayton58/43/orig 2025-12-04T08:57:44.4242743Z * [new branch] gh/slayton58/44/base -> origin/gh/slayton58/44/base 2025-12-04T08:57:44.4243852Z * [new branch] gh/slayton58/44/head -> origin/gh/slayton58/44/head 2025-12-04T08:57:44.4245109Z * [new branch] gh/slayton58/44/orig -> origin/gh/slayton58/44/orig 2025-12-04T08:57:44.4246532Z * [new branch] gh/slayton58/45/base -> origin/gh/slayton58/45/base 2025-12-04T08:57:44.4247753Z * [new branch] gh/slayton58/45/head -> origin/gh/slayton58/45/head 2025-12-04T08:57:44.4248889Z * [new branch] gh/slayton58/45/orig -> origin/gh/slayton58/45/orig 2025-12-04T08:57:44.4250840Z * [new branch] gh/slayton58/46/base -> origin/gh/slayton58/46/base 2025-12-04T08:57:44.4252073Z * [new branch] gh/slayton58/46/head -> origin/gh/slayton58/46/head 2025-12-04T08:57:44.4253249Z * [new branch] gh/slayton58/46/orig -> origin/gh/slayton58/46/orig 2025-12-04T08:57:44.4254709Z * [new branch] gh/slayton58/6/base -> origin/gh/slayton58/6/base 2025-12-04T08:57:44.4255908Z * [new branch] gh/slayton58/6/head -> origin/gh/slayton58/6/head 2025-12-04T08:57:44.4257222Z * [new branch] gh/slayton58/7/base -> origin/gh/slayton58/7/base 2025-12-04T08:57:44.4258285Z * [new branch] gh/slayton58/7/head -> origin/gh/slayton58/7/head 2025-12-04T08:57:44.4261170Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-12-04T08:57:44.4262241Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-12-04T08:57:44.4263476Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-12-04T08:57:44.4265056Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-12-04T08:57:44.4266177Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-12-04T08:57:44.4267336Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-12-04T08:57:44.4269484Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-12-04T08:57:44.4270567Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-12-04T08:57:44.4271750Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-12-04T08:57:44.4273429Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-12-04T08:57:44.4274577Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-12-04T08:57:44.4275698Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-12-04T08:57:44.4277429Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-12-04T08:57:44.4278648Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-12-04T08:57:44.4279900Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-12-04T08:57:44.4281378Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-12-04T08:57:44.4282608Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-12-04T08:57:44.4283667Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-12-04T08:57:44.4285824Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-12-04T08:57:44.4286952Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-12-04T08:57:44.4288053Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-12-04T08:57:44.4289507Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-12-04T08:57:44.4290596Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-12-04T08:57:44.4291743Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-12-04T08:57:44.4293364Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-12-04T08:57:44.4294414Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-12-04T08:57:44.4295470Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-12-04T08:57:44.4297112Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-12-04T08:57:44.4298201Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-12-04T08:57:44.4299262Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-12-04T08:57:44.4300868Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-12-04T08:57:44.4301912Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-12-04T08:57:44.4302972Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-12-04T08:57:44.4304493Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-12-04T08:57:44.4305542Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-12-04T08:57:44.4306633Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-12-04T08:57:44.4308774Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-12-04T08:57:44.4310000Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-12-04T08:57:44.4311120Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-12-04T08:57:44.4312543Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-12-04T08:57:44.4313606Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-12-04T08:57:44.4314776Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-12-04T08:57:44.4316381Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-12-04T08:57:44.4317479Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-12-04T08:57:44.4318608Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-12-04T08:57:44.4320235Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-12-04T08:57:44.4321435Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-12-04T08:57:44.4322523Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-12-04T08:57:44.4325333Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-12-04T08:57:44.4326535Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-12-04T08:57:44.4327675Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-12-04T08:57:44.4330162Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-12-04T08:57:44.4331331Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-12-04T08:57:44.4332627Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-12-04T08:57:44.4334198Z * [new branch] gh/soulitzer/374/base -> origin/gh/soulitzer/374/base 2025-12-04T08:57:44.4335357Z * [new branch] gh/soulitzer/374/head -> origin/gh/soulitzer/374/head 2025-12-04T08:57:44.4336558Z * [new branch] gh/soulitzer/374/orig -> origin/gh/soulitzer/374/orig 2025-12-04T08:57:44.4338027Z * [new branch] gh/soulitzer/375/base -> origin/gh/soulitzer/375/base 2025-12-04T08:57:44.4339059Z * [new branch] gh/soulitzer/375/head -> origin/gh/soulitzer/375/head 2025-12-04T08:57:44.4340296Z * [new branch] gh/soulitzer/375/orig -> origin/gh/soulitzer/375/orig 2025-12-04T08:57:44.4341822Z * [new branch] gh/soulitzer/380/base -> origin/gh/soulitzer/380/base 2025-12-04T08:57:44.4342901Z * [new branch] gh/soulitzer/380/head -> origin/gh/soulitzer/380/head 2025-12-04T08:57:44.4344036Z * [new branch] gh/soulitzer/380/orig -> origin/gh/soulitzer/380/orig 2025-12-04T08:57:44.4345496Z * [new branch] gh/soulitzer/385/base -> origin/gh/soulitzer/385/base 2025-12-04T08:57:44.4346576Z * [new branch] gh/soulitzer/385/head -> origin/gh/soulitzer/385/head 2025-12-04T08:57:44.4347908Z * [new branch] gh/soulitzer/385/orig -> origin/gh/soulitzer/385/orig 2025-12-04T08:57:44.4349508Z * [new branch] gh/soulitzer/386/base -> origin/gh/soulitzer/386/base 2025-12-04T08:57:44.4350657Z * [new branch] gh/soulitzer/386/head -> origin/gh/soulitzer/386/head 2025-12-04T08:57:44.4351761Z * [new branch] gh/soulitzer/386/orig -> origin/gh/soulitzer/386/orig 2025-12-04T08:57:44.4353273Z * [new branch] gh/soulitzer/387/base -> origin/gh/soulitzer/387/base 2025-12-04T08:57:44.4354378Z * [new branch] gh/soulitzer/387/head -> origin/gh/soulitzer/387/head 2025-12-04T08:57:44.4355509Z * [new branch] gh/soulitzer/387/orig -> origin/gh/soulitzer/387/orig 2025-12-04T08:57:44.4357146Z * [new branch] gh/soulitzer/388/base -> origin/gh/soulitzer/388/base 2025-12-04T08:57:44.4358294Z * [new branch] gh/soulitzer/388/head -> origin/gh/soulitzer/388/head 2025-12-04T08:57:44.4359430Z * [new branch] gh/soulitzer/388/orig -> origin/gh/soulitzer/388/orig 2025-12-04T08:57:44.4361057Z * [new branch] gh/soulitzer/389/base -> origin/gh/soulitzer/389/base 2025-12-04T08:57:44.4362122Z * [new branch] gh/soulitzer/389/head -> origin/gh/soulitzer/389/head 2025-12-04T08:57:44.4363243Z * [new branch] gh/soulitzer/389/orig -> origin/gh/soulitzer/389/orig 2025-12-04T08:57:44.4364679Z * [new branch] gh/soulitzer/390/base -> origin/gh/soulitzer/390/base 2025-12-04T08:57:44.4365785Z * [new branch] gh/soulitzer/390/head -> origin/gh/soulitzer/390/head 2025-12-04T08:57:44.4366875Z * [new branch] gh/soulitzer/390/orig -> origin/gh/soulitzer/390/orig 2025-12-04T08:57:44.4368328Z * [new branch] gh/soulitzer/391/base -> origin/gh/soulitzer/391/base 2025-12-04T08:57:44.4369479Z * [new branch] gh/soulitzer/391/head -> origin/gh/soulitzer/391/head 2025-12-04T08:57:44.4370550Z * [new branch] gh/soulitzer/391/orig -> origin/gh/soulitzer/391/orig 2025-12-04T08:57:44.4372164Z * [new branch] gh/soulitzer/392/base -> origin/gh/soulitzer/392/base 2025-12-04T08:57:44.4373260Z * [new branch] gh/soulitzer/392/head -> origin/gh/soulitzer/392/head 2025-12-04T08:57:44.4374339Z * [new branch] gh/soulitzer/392/orig -> origin/gh/soulitzer/392/orig 2025-12-04T08:57:44.4376070Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-12-04T08:57:44.4377822Z * [new branch] gh/swolchok/819/base -> origin/gh/swolchok/819/base 2025-12-04T08:57:44.4378942Z * [new branch] gh/swolchok/819/head -> origin/gh/swolchok/819/head 2025-12-04T08:57:44.4379993Z * [new branch] gh/swolchok/819/orig -> origin/gh/swolchok/819/orig 2025-12-04T08:57:44.4381450Z * [new branch] gh/swolchok/824/base -> origin/gh/swolchok/824/base 2025-12-04T08:57:44.4382668Z * [new branch] gh/swolchok/824/head -> origin/gh/swolchok/824/head 2025-12-04T08:57:44.4383791Z * [new branch] gh/swolchok/824/orig -> origin/gh/swolchok/824/orig 2025-12-04T08:57:44.4385436Z * [new branch] gh/swolchok/829/base -> origin/gh/swolchok/829/base 2025-12-04T08:57:44.4386943Z * [new branch] gh/swolchok/829/head -> origin/gh/swolchok/829/head 2025-12-04T08:57:44.4388429Z * [new branch] gh/swolchok/829/orig -> origin/gh/swolchok/829/orig 2025-12-04T08:57:44.4390018Z * [new branch] gh/swolchok/839/base -> origin/gh/swolchok/839/base 2025-12-04T08:57:44.4391083Z * [new branch] gh/swolchok/839/head -> origin/gh/swolchok/839/head 2025-12-04T08:57:44.4392208Z * [new branch] gh/swolchok/839/orig -> origin/gh/swolchok/839/orig 2025-12-04T08:57:44.4393709Z * [new branch] gh/swolchok/841/base -> origin/gh/swolchok/841/base 2025-12-04T08:57:44.4394852Z * [new branch] gh/swolchok/841/head -> origin/gh/swolchok/841/head 2025-12-04T08:57:44.4396073Z * [new branch] gh/swolchok/841/orig -> origin/gh/swolchok/841/orig 2025-12-04T08:57:44.4397581Z * [new branch] gh/swolchok/842/base -> origin/gh/swolchok/842/base 2025-12-04T08:57:44.4398737Z * [new branch] gh/swolchok/842/head -> origin/gh/swolchok/842/head 2025-12-04T08:57:44.4399962Z * [new branch] gh/swolchok/842/orig -> origin/gh/swolchok/842/orig 2025-12-04T08:57:44.4401397Z * [new branch] gh/swolchok/845/base -> origin/gh/swolchok/845/base 2025-12-04T08:57:44.4402595Z * [new branch] gh/swolchok/845/head -> origin/gh/swolchok/845/head 2025-12-04T08:57:44.4403781Z * [new branch] gh/swolchok/845/orig -> origin/gh/swolchok/845/orig 2025-12-04T08:57:44.4405332Z * [new branch] gh/swolchok/848/base -> origin/gh/swolchok/848/base 2025-12-04T08:57:44.4406582Z * [new branch] gh/swolchok/848/head -> origin/gh/swolchok/848/head 2025-12-04T08:57:44.4407702Z * [new branch] gh/swolchok/848/orig -> origin/gh/swolchok/848/orig 2025-12-04T08:57:44.4409205Z * [new branch] gh/swolchok/856/base -> origin/gh/swolchok/856/base 2025-12-04T08:57:44.4410290Z * [new branch] gh/swolchok/856/head -> origin/gh/swolchok/856/head 2025-12-04T08:57:44.4411389Z * [new branch] gh/swolchok/856/orig -> origin/gh/swolchok/856/orig 2025-12-04T08:57:44.4412954Z * [new branch] gh/swolchok/860/base -> origin/gh/swolchok/860/base 2025-12-04T08:57:44.4414036Z * [new branch] gh/swolchok/860/head -> origin/gh/swolchok/860/head 2025-12-04T08:57:44.4415075Z * [new branch] gh/swolchok/860/orig -> origin/gh/swolchok/860/orig 2025-12-04T08:57:44.4416745Z * [new branch] gh/swolchok/861/base -> origin/gh/swolchok/861/base 2025-12-04T08:57:44.4418011Z * [new branch] gh/swolchok/861/head -> origin/gh/swolchok/861/head 2025-12-04T08:57:44.4419162Z * [new branch] gh/swolchok/861/orig -> origin/gh/swolchok/861/orig 2025-12-04T08:57:44.4420729Z * [new branch] gh/swolchok/862/base -> origin/gh/swolchok/862/base 2025-12-04T08:57:44.4421804Z * [new branch] gh/swolchok/862/head -> origin/gh/swolchok/862/head 2025-12-04T08:57:44.4422869Z * [new branch] gh/swolchok/862/orig -> origin/gh/swolchok/862/orig 2025-12-04T08:57:44.4425027Z * [new branch] gh/swolchok/863/base -> origin/gh/swolchok/863/base 2025-12-04T08:57:44.4426572Z * [new branch] gh/swolchok/863/head -> origin/gh/swolchok/863/head 2025-12-04T08:57:44.4427964Z * [new branch] gh/swolchok/863/orig -> origin/gh/swolchok/863/orig 2025-12-04T08:57:44.4430127Z * [new branch] gh/swolchok/864/base -> origin/gh/swolchok/864/base 2025-12-04T08:57:44.4431205Z * [new branch] gh/swolchok/864/head -> origin/gh/swolchok/864/head 2025-12-04T08:57:44.4432634Z * [new branch] gh/swolchok/864/orig -> origin/gh/swolchok/864/orig 2025-12-04T08:57:44.4433917Z * [new branch] gh/swolchok/865/base -> origin/gh/swolchok/865/base 2025-12-04T08:57:44.4435455Z * [new branch] gh/swolchok/865/head -> origin/gh/swolchok/865/head 2025-12-04T08:57:44.4436577Z * [new branch] gh/swolchok/865/orig -> origin/gh/swolchok/865/orig 2025-12-04T08:57:44.4438806Z * [new branch] gh/swolchok/866/base -> origin/gh/swolchok/866/base 2025-12-04T08:57:44.4439992Z * [new branch] gh/swolchok/866/head -> origin/gh/swolchok/866/head 2025-12-04T08:57:44.4441125Z * [new branch] gh/swolchok/866/orig -> origin/gh/swolchok/866/orig 2025-12-04T08:57:44.4442659Z * [new branch] gh/swolchok/867/base -> origin/gh/swolchok/867/base 2025-12-04T08:57:44.4443694Z * [new branch] gh/swolchok/867/head -> origin/gh/swolchok/867/head 2025-12-04T08:57:44.4445161Z * [new branch] gh/swolchok/867/orig -> origin/gh/swolchok/867/orig 2025-12-04T08:57:44.4446732Z * [new branch] gh/swolchok/868/base -> origin/gh/swolchok/868/base 2025-12-04T08:57:44.4447853Z * [new branch] gh/swolchok/868/head -> origin/gh/swolchok/868/head 2025-12-04T08:57:44.4448933Z * [new branch] gh/swolchok/868/orig -> origin/gh/swolchok/868/orig 2025-12-04T08:57:44.4450432Z * [new branch] gh/swolchok/869/base -> origin/gh/swolchok/869/base 2025-12-04T08:57:44.4451845Z * [new branch] gh/swolchok/869/head -> origin/gh/swolchok/869/head 2025-12-04T08:57:44.4452926Z * [new branch] gh/swolchok/869/orig -> origin/gh/swolchok/869/orig 2025-12-04T08:57:44.4454531Z * [new branch] gh/swolchok/870/base -> origin/gh/swolchok/870/base 2025-12-04T08:57:44.4455993Z * [new branch] gh/swolchok/870/head -> origin/gh/swolchok/870/head 2025-12-04T08:57:44.4457538Z * [new branch] gh/swolchok/870/orig -> origin/gh/swolchok/870/orig 2025-12-04T08:57:44.4459057Z * [new branch] gh/swolchok/871/base -> origin/gh/swolchok/871/base 2025-12-04T08:57:44.4460249Z * [new branch] gh/swolchok/871/head -> origin/gh/swolchok/871/head 2025-12-04T08:57:44.4461432Z * [new branch] gh/swolchok/871/orig -> origin/gh/swolchok/871/orig 2025-12-04T08:57:44.4463298Z * [new branch] gh/teja-rao/4/base -> origin/gh/teja-rao/4/base 2025-12-04T08:57:44.4464442Z * [new branch] gh/teja-rao/4/head -> origin/gh/teja-rao/4/head 2025-12-04T08:57:44.4465573Z * [new branch] gh/teja-rao/4/orig -> origin/gh/teja-rao/4/orig 2025-12-04T08:57:44.4467471Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-12-04T08:57:44.4468937Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-12-04T08:57:44.4470029Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-12-04T08:57:44.4471543Z * [new branch] gh/tianyu-l/3/base -> origin/gh/tianyu-l/3/base 2025-12-04T08:57:44.4472698Z * [new branch] gh/tianyu-l/3/orig -> origin/gh/tianyu-l/3/orig 2025-12-04T08:57:44.4474250Z * [new branch] gh/tianyu-l/4/base -> origin/gh/tianyu-l/4/base 2025-12-04T08:57:44.4475320Z * [new branch] gh/tianyu-l/4/head -> origin/gh/tianyu-l/4/head 2025-12-04T08:57:44.4476510Z * [new branch] gh/tianyu-l/4/orig -> origin/gh/tianyu-l/4/orig 2025-12-04T08:57:44.4478842Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2025-12-04T08:57:44.4480078Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2025-12-04T08:57:44.4481426Z * [new branch] gh/tugsbayasgalan/10/orig -> origin/gh/tugsbayasgalan/10/orig 2025-12-04T08:57:44.4482855Z * [new branch] gh/tugsbayasgalan/13/base -> origin/gh/tugsbayasgalan/13/base 2025-12-04T08:57:44.4484019Z * [new branch] gh/tugsbayasgalan/13/head -> origin/gh/tugsbayasgalan/13/head 2025-12-04T08:57:44.4485142Z * [new branch] gh/tugsbayasgalan/13/orig -> origin/gh/tugsbayasgalan/13/orig 2025-12-04T08:57:44.4486651Z * [new branch] gh/tugsbayasgalan/17/base -> origin/gh/tugsbayasgalan/17/base 2025-12-04T08:57:44.4487702Z * [new branch] gh/tugsbayasgalan/17/head -> origin/gh/tugsbayasgalan/17/head 2025-12-04T08:57:44.4488774Z * [new branch] gh/tugsbayasgalan/17/orig -> origin/gh/tugsbayasgalan/17/orig 2025-12-04T08:57:44.4490345Z * [new branch] gh/tugsbayasgalan/2/base -> origin/gh/tugsbayasgalan/2/base 2025-12-04T08:57:44.4491450Z * [new branch] gh/tugsbayasgalan/2/head -> origin/gh/tugsbayasgalan/2/head 2025-12-04T08:57:44.4492547Z * [new branch] gh/tugsbayasgalan/2/orig -> origin/gh/tugsbayasgalan/2/orig 2025-12-04T08:57:44.4494325Z * [new branch] gh/tugsbayasgalan/28/base -> origin/gh/tugsbayasgalan/28/base 2025-12-04T08:57:44.4495411Z * [new branch] gh/tugsbayasgalan/28/head -> origin/gh/tugsbayasgalan/28/head 2025-12-04T08:57:44.4496529Z * [new branch] gh/tugsbayasgalan/28/orig -> origin/gh/tugsbayasgalan/28/orig 2025-12-04T08:57:44.4498572Z * [new branch] gh/tugsbayasgalan/32/base -> origin/gh/tugsbayasgalan/32/base 2025-12-04T08:57:44.4499725Z * [new branch] gh/tugsbayasgalan/32/head -> origin/gh/tugsbayasgalan/32/head 2025-12-04T08:57:44.4500869Z * [new branch] gh/tugsbayasgalan/32/orig -> origin/gh/tugsbayasgalan/32/orig 2025-12-04T08:57:44.4502487Z * [new branch] gh/tugsbayasgalan/35/base -> origin/gh/tugsbayasgalan/35/base 2025-12-04T08:57:44.4503629Z * [new branch] gh/tugsbayasgalan/35/head -> origin/gh/tugsbayasgalan/35/head 2025-12-04T08:57:44.4504779Z * [new branch] gh/tugsbayasgalan/35/orig -> origin/gh/tugsbayasgalan/35/orig 2025-12-04T08:57:44.4506261Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2025-12-04T08:57:44.4507383Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2025-12-04T08:57:44.4508810Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2025-12-04T08:57:44.4510324Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2025-12-04T08:57:44.4511472Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2025-12-04T08:57:44.4512581Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2025-12-04T08:57:44.4514645Z * [new branch] gh/tugsbayasgalan/43/base -> origin/gh/tugsbayasgalan/43/base 2025-12-04T08:57:44.4516238Z * [new branch] gh/tugsbayasgalan/43/head -> origin/gh/tugsbayasgalan/43/head 2025-12-04T08:57:44.4517332Z * [new branch] gh/tugsbayasgalan/43/orig -> origin/gh/tugsbayasgalan/43/orig 2025-12-04T08:57:44.4518787Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2025-12-04T08:57:44.4520073Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2025-12-04T08:57:44.4521396Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2025-12-04T08:57:44.4522893Z * [new branch] gh/tugsbayasgalan/51/base -> origin/gh/tugsbayasgalan/51/base 2025-12-04T08:57:44.4528197Z * [new branch] gh/tugsbayasgalan/51/head -> origin/gh/tugsbayasgalan/51/head 2025-12-04T08:57:44.4529622Z * [new branch] gh/tugsbayasgalan/51/orig -> origin/gh/tugsbayasgalan/51/orig 2025-12-04T08:57:44.4531343Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2025-12-04T08:57:44.4532551Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2025-12-04T08:57:44.4533676Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2025-12-04T08:57:44.4535485Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2025-12-04T08:57:44.4536690Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2025-12-04T08:57:44.4537767Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2025-12-04T08:57:44.4557839Z * [new branch] gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base 2025-12-04T08:57:44.4558298Z * [new branch] gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head 2025-12-04T08:57:44.4558615Z * [new branch] gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig 2025-12-04T08:57:44.4558911Z * [new branch] gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base 2025-12-04T08:57:44.4559210Z * [new branch] gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head 2025-12-04T08:57:44.4559495Z * [new branch] gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig 2025-12-04T08:57:44.4559889Z * [new branch] gh/tugsbayasgalan/6/base -> origin/gh/tugsbayasgalan/6/base 2025-12-04T08:57:44.4560181Z * [new branch] gh/tugsbayasgalan/6/head -> origin/gh/tugsbayasgalan/6/head 2025-12-04T08:57:44.4560454Z * [new branch] gh/tugsbayasgalan/6/orig -> origin/gh/tugsbayasgalan/6/orig 2025-12-04T08:57:44.4560747Z * [new branch] gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base 2025-12-04T08:57:44.4561028Z * [new branch] gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head 2025-12-04T08:57:44.4561307Z * [new branch] gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig 2025-12-04T08:57:44.4561595Z * [new branch] gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base 2025-12-04T08:57:44.4561868Z * [new branch] gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head 2025-12-04T08:57:44.4562157Z * [new branch] gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig 2025-12-04T08:57:44.4562436Z * [new branch] gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base 2025-12-04T08:57:44.4562708Z * [new branch] gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head 2025-12-04T08:57:44.4562993Z * [new branch] gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig 2025-12-04T08:57:44.4563706Z * [new branch] gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base 2025-12-04T08:57:44.4564850Z * [new branch] gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head 2025-12-04T08:57:44.4565950Z * [new branch] gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig 2025-12-04T08:57:44.4567760Z * [new branch] gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base 2025-12-04T08:57:44.4568888Z * [new branch] gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head 2025-12-04T08:57:44.4569993Z * [new branch] gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig 2025-12-04T08:57:44.4571539Z * [new branch] gh/tugsbayasgalan/7/base -> origin/gh/tugsbayasgalan/7/base 2025-12-04T08:57:44.4572645Z * [new branch] gh/tugsbayasgalan/7/head -> origin/gh/tugsbayasgalan/7/head 2025-12-04T08:57:44.4573747Z * [new branch] gh/tugsbayasgalan/7/orig -> origin/gh/tugsbayasgalan/7/orig 2025-12-04T08:57:44.4575651Z * [new branch] gh/tugsbayasgalan/70/base -> origin/gh/tugsbayasgalan/70/base 2025-12-04T08:57:44.4576842Z * [new branch] gh/tugsbayasgalan/70/head -> origin/gh/tugsbayasgalan/70/head 2025-12-04T08:57:44.4577971Z * [new branch] gh/tugsbayasgalan/70/orig -> origin/gh/tugsbayasgalan/70/orig 2025-12-04T08:57:44.4579609Z * [new branch] gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base 2025-12-04T08:57:44.4580895Z * [new branch] gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head 2025-12-04T08:57:44.4582011Z * [new branch] gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig 2025-12-04T08:57:44.4584282Z * [new branch] gh/tugsbayasgalan/72/base -> origin/gh/tugsbayasgalan/72/base 2025-12-04T08:57:44.4585419Z * [new branch] gh/tugsbayasgalan/72/head -> origin/gh/tugsbayasgalan/72/head 2025-12-04T08:57:44.4586536Z * [new branch] gh/tugsbayasgalan/72/orig -> origin/gh/tugsbayasgalan/72/orig 2025-12-04T08:57:44.4588868Z * [new branch] gh/tugsbayasgalan/73/base -> origin/gh/tugsbayasgalan/73/base 2025-12-04T08:57:44.4590122Z * [new branch] gh/tugsbayasgalan/73/head -> origin/gh/tugsbayasgalan/73/head 2025-12-04T08:57:44.4591317Z * [new branch] gh/tugsbayasgalan/73/orig -> origin/gh/tugsbayasgalan/73/orig 2025-12-04T08:57:44.4593030Z * [new branch] gh/tugsbayasgalan/74/base -> origin/gh/tugsbayasgalan/74/base 2025-12-04T08:57:44.4594259Z * [new branch] gh/tugsbayasgalan/74/head -> origin/gh/tugsbayasgalan/74/head 2025-12-04T08:57:44.4595587Z * [new branch] gh/tugsbayasgalan/74/orig -> origin/gh/tugsbayasgalan/74/orig 2025-12-04T08:57:44.4597186Z * [new branch] gh/tugsbayasgalan/75/base -> origin/gh/tugsbayasgalan/75/base 2025-12-04T08:57:44.4598397Z * [new branch] gh/tugsbayasgalan/75/head -> origin/gh/tugsbayasgalan/75/head 2025-12-04T08:57:44.4599521Z * [new branch] gh/tugsbayasgalan/75/orig -> origin/gh/tugsbayasgalan/75/orig 2025-12-04T08:57:44.4601124Z * [new branch] gh/tugsbayasgalan/76/base -> origin/gh/tugsbayasgalan/76/base 2025-12-04T08:57:44.4602219Z * [new branch] gh/tugsbayasgalan/76/head -> origin/gh/tugsbayasgalan/76/head 2025-12-04T08:57:44.4603339Z * [new branch] gh/tugsbayasgalan/76/orig -> origin/gh/tugsbayasgalan/76/orig 2025-12-04T08:57:44.4605030Z * [new branch] gh/tugsbayasgalan/77/base -> origin/gh/tugsbayasgalan/77/base 2025-12-04T08:57:44.4606080Z * [new branch] gh/tugsbayasgalan/77/head -> origin/gh/tugsbayasgalan/77/head 2025-12-04T08:57:44.4607166Z * [new branch] gh/tugsbayasgalan/77/orig -> origin/gh/tugsbayasgalan/77/orig 2025-12-04T08:57:44.4608714Z * [new branch] gh/tugsbayasgalan/78/base -> origin/gh/tugsbayasgalan/78/base 2025-12-04T08:57:44.4610001Z * [new branch] gh/tugsbayasgalan/78/head -> origin/gh/tugsbayasgalan/78/head 2025-12-04T08:57:44.4611098Z * [new branch] gh/tugsbayasgalan/78/orig -> origin/gh/tugsbayasgalan/78/orig 2025-12-04T08:57:44.4612621Z * [new branch] gh/tugsbayasgalan/79/base -> origin/gh/tugsbayasgalan/79/base 2025-12-04T08:57:44.4613708Z * [new branch] gh/tugsbayasgalan/79/head -> origin/gh/tugsbayasgalan/79/head 2025-12-04T08:57:44.4614809Z * [new branch] gh/tugsbayasgalan/79/orig -> origin/gh/tugsbayasgalan/79/orig 2025-12-04T08:57:44.4616406Z * [new branch] gh/tugsbayasgalan/8/base -> origin/gh/tugsbayasgalan/8/base 2025-12-04T08:57:44.4617450Z * [new branch] gh/tugsbayasgalan/8/head -> origin/gh/tugsbayasgalan/8/head 2025-12-04T08:57:44.4618606Z * [new branch] gh/tugsbayasgalan/8/orig -> origin/gh/tugsbayasgalan/8/orig 2025-12-04T08:57:44.4620095Z * [new branch] gh/tugsbayasgalan/80/base -> origin/gh/tugsbayasgalan/80/base 2025-12-04T08:57:44.4620979Z * [new branch] gh/tugsbayasgalan/80/head -> origin/gh/tugsbayasgalan/80/head 2025-12-04T08:57:44.4622055Z * [new branch] gh/tugsbayasgalan/80/orig -> origin/gh/tugsbayasgalan/80/orig 2025-12-04T08:57:44.4623741Z * [new branch] gh/tugsbayasgalan/81/base -> origin/gh/tugsbayasgalan/81/base 2025-12-04T08:57:44.4625076Z * [new branch] gh/tugsbayasgalan/81/head -> origin/gh/tugsbayasgalan/81/head 2025-12-04T08:57:44.4626163Z * [new branch] gh/tugsbayasgalan/81/orig -> origin/gh/tugsbayasgalan/81/orig 2025-12-04T08:57:44.4628518Z * [new branch] gh/tugsbayasgalan/82/base -> origin/gh/tugsbayasgalan/82/base 2025-12-04T08:57:44.4629832Z * [new branch] gh/tugsbayasgalan/82/head -> origin/gh/tugsbayasgalan/82/head 2025-12-04T08:57:44.4631043Z * [new branch] gh/tugsbayasgalan/82/orig -> origin/gh/tugsbayasgalan/82/orig 2025-12-04T08:57:44.4632626Z * [new branch] gh/tugsbayasgalan/83/base -> origin/gh/tugsbayasgalan/83/base 2025-12-04T08:57:44.4633753Z * [new branch] gh/tugsbayasgalan/83/head -> origin/gh/tugsbayasgalan/83/head 2025-12-04T08:57:44.4634881Z * [new branch] gh/tugsbayasgalan/83/orig -> origin/gh/tugsbayasgalan/83/orig 2025-12-04T08:57:44.4636288Z * [new branch] gh/tugsbayasgalan/84/base -> origin/gh/tugsbayasgalan/84/base 2025-12-04T08:57:44.4637415Z * [new branch] gh/tugsbayasgalan/84/head -> origin/gh/tugsbayasgalan/84/head 2025-12-04T08:57:44.4638552Z * [new branch] gh/tugsbayasgalan/84/orig -> origin/gh/tugsbayasgalan/84/orig 2025-12-04T08:57:44.4640049Z * [new branch] gh/tugsbayasgalan/85/base -> origin/gh/tugsbayasgalan/85/base 2025-12-04T08:57:44.4641144Z * [new branch] gh/tugsbayasgalan/85/head -> origin/gh/tugsbayasgalan/85/head 2025-12-04T08:57:44.4642654Z * [new branch] gh/tugsbayasgalan/85/orig -> origin/gh/tugsbayasgalan/85/orig 2025-12-04T08:57:44.4644204Z * [new branch] gh/tugsbayasgalan/86/base -> origin/gh/tugsbayasgalan/86/base 2025-12-04T08:57:44.4645339Z * [new branch] gh/tugsbayasgalan/86/head -> origin/gh/tugsbayasgalan/86/head 2025-12-04T08:57:44.4646463Z * [new branch] gh/tugsbayasgalan/86/orig -> origin/gh/tugsbayasgalan/86/orig 2025-12-04T08:57:44.4648389Z * [new branch] gh/tugsbayasgalan/87/base -> origin/gh/tugsbayasgalan/87/base 2025-12-04T08:57:44.4649515Z * [new branch] gh/tugsbayasgalan/87/head -> origin/gh/tugsbayasgalan/87/head 2025-12-04T08:57:44.4650575Z * [new branch] gh/tugsbayasgalan/87/orig -> origin/gh/tugsbayasgalan/87/orig 2025-12-04T08:57:44.4652133Z * [new branch] gh/tugsbayasgalan/88/base -> origin/gh/tugsbayasgalan/88/base 2025-12-04T08:57:44.4653209Z * [new branch] gh/tugsbayasgalan/88/head -> origin/gh/tugsbayasgalan/88/head 2025-12-04T08:57:44.4654345Z * [new branch] gh/tugsbayasgalan/88/orig -> origin/gh/tugsbayasgalan/88/orig 2025-12-04T08:57:44.4656323Z * [new branch] gh/tugsbayasgalan/89/base -> origin/gh/tugsbayasgalan/89/base 2025-12-04T08:57:44.4657548Z * [new branch] gh/tugsbayasgalan/89/head -> origin/gh/tugsbayasgalan/89/head 2025-12-04T08:57:44.4658639Z * [new branch] gh/tugsbayasgalan/89/orig -> origin/gh/tugsbayasgalan/89/orig 2025-12-04T08:57:44.4660102Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2025-12-04T08:57:44.4661072Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2025-12-04T08:57:44.4662286Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2025-12-04T08:57:44.4664008Z * [new branch] gh/tugsbayasgalan/90/base -> origin/gh/tugsbayasgalan/90/base 2025-12-04T08:57:44.4665196Z * [new branch] gh/tugsbayasgalan/90/head -> origin/gh/tugsbayasgalan/90/head 2025-12-04T08:57:44.4666121Z * [new branch] gh/tugsbayasgalan/90/orig -> origin/gh/tugsbayasgalan/90/orig 2025-12-04T08:57:44.4668045Z * [new branch] gh/tugsbayasgalan/91/base -> origin/gh/tugsbayasgalan/91/base 2025-12-04T08:57:44.4669141Z * [new branch] gh/tugsbayasgalan/91/head -> origin/gh/tugsbayasgalan/91/head 2025-12-04T08:57:44.4670339Z * [new branch] gh/tugsbayasgalan/91/orig -> origin/gh/tugsbayasgalan/91/orig 2025-12-04T08:57:44.4672023Z * [new branch] gh/tugsbayasgalan/92/base -> origin/gh/tugsbayasgalan/92/base 2025-12-04T08:57:44.4673149Z * [new branch] gh/tugsbayasgalan/92/head -> origin/gh/tugsbayasgalan/92/head 2025-12-04T08:57:44.4674285Z * [new branch] gh/tugsbayasgalan/92/orig -> origin/gh/tugsbayasgalan/92/orig 2025-12-04T08:57:44.4675953Z * [new branch] gh/tugsbayasgalan/93/base -> origin/gh/tugsbayasgalan/93/base 2025-12-04T08:57:44.4677164Z * [new branch] gh/tugsbayasgalan/93/head -> origin/gh/tugsbayasgalan/93/head 2025-12-04T08:57:44.4678318Z * [new branch] gh/tugsbayasgalan/93/orig -> origin/gh/tugsbayasgalan/93/orig 2025-12-04T08:57:44.4680350Z * [new branch] gh/v0i0/14/base -> origin/gh/v0i0/14/base 2025-12-04T08:57:44.4681401Z * [new branch] gh/v0i0/14/head -> origin/gh/v0i0/14/head 2025-12-04T08:57:44.4682460Z * [new branch] gh/v0i0/14/orig -> origin/gh/v0i0/14/orig 2025-12-04T08:57:44.4683765Z * [new branch] gh/v0i0/15/base -> origin/gh/v0i0/15/base 2025-12-04T08:57:44.4684951Z * [new branch] gh/v0i0/15/head -> origin/gh/v0i0/15/head 2025-12-04T08:57:44.4686080Z * [new branch] gh/v0i0/15/orig -> origin/gh/v0i0/15/orig 2025-12-04T08:57:44.4687595Z * [new branch] gh/v0i0/16/base -> origin/gh/v0i0/16/base 2025-12-04T08:57:44.4688655Z * [new branch] gh/v0i0/16/head -> origin/gh/v0i0/16/head 2025-12-04T08:57:44.4689783Z * [new branch] gh/v0i0/16/orig -> origin/gh/v0i0/16/orig 2025-12-04T08:57:44.4691192Z * [new branch] gh/v0i0/17/base -> origin/gh/v0i0/17/base 2025-12-04T08:57:44.4692302Z * [new branch] gh/v0i0/17/head -> origin/gh/v0i0/17/head 2025-12-04T08:57:44.4693466Z * [new branch] gh/v0i0/17/orig -> origin/gh/v0i0/17/orig 2025-12-04T08:57:44.4694951Z * [new branch] gh/v0i0/18/base -> origin/gh/v0i0/18/base 2025-12-04T08:57:44.4696115Z * [new branch] gh/v0i0/18/head -> origin/gh/v0i0/18/head 2025-12-04T08:57:44.4697179Z * [new branch] gh/v0i0/18/orig -> origin/gh/v0i0/18/orig 2025-12-04T08:57:44.4698722Z * [new branch] gh/v0i0/19/base -> origin/gh/v0i0/19/base 2025-12-04T08:57:44.4699824Z * [new branch] gh/v0i0/19/head -> origin/gh/v0i0/19/head 2025-12-04T08:57:44.4700946Z * [new branch] gh/v0i0/19/orig -> origin/gh/v0i0/19/orig 2025-12-04T08:57:44.4702723Z * [new branch] gh/vishal9-team/1/base -> origin/gh/vishal9-team/1/base 2025-12-04T08:57:44.4703833Z * [new branch] gh/vishal9-team/1/head -> origin/gh/vishal9-team/1/head 2025-12-04T08:57:44.4705236Z * [new branch] gh/vishal9-team/2/base -> origin/gh/vishal9-team/2/base 2025-12-04T08:57:44.4706339Z * [new branch] gh/vishal9-team/2/head -> origin/gh/vishal9-team/2/head 2025-12-04T08:57:44.4707654Z * [new branch] gh/vishal9-team/2/orig -> origin/gh/vishal9-team/2/orig 2025-12-04T08:57:44.4709402Z * [new branch] gh/vishal9-team/3/base -> origin/gh/vishal9-team/3/base 2025-12-04T08:57:44.4710683Z * [new branch] gh/vishal9-team/3/head -> origin/gh/vishal9-team/3/head 2025-12-04T08:57:44.4711729Z * [new branch] gh/vishal9-team/3/orig -> origin/gh/vishal9-team/3/orig 2025-12-04T08:57:44.4713109Z * [new branch] gh/vishal9-team/4/base -> origin/gh/vishal9-team/4/base 2025-12-04T08:57:44.4714244Z * [new branch] gh/vishal9-team/4/head -> origin/gh/vishal9-team/4/head 2025-12-04T08:57:44.4715358Z * [new branch] gh/vishal9-team/4/orig -> origin/gh/vishal9-team/4/orig 2025-12-04T08:57:44.4717085Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-12-04T08:57:44.4718557Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-12-04T08:57:44.4720139Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-12-04T08:57:44.4722033Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-12-04T08:57:44.4723221Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-12-04T08:57:44.4724762Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-12-04T08:57:44.4726297Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-12-04T08:57:44.4727591Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-12-04T08:57:44.4728874Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-12-04T08:57:44.4730402Z * [new branch] gh/wconstab/444/base -> origin/gh/wconstab/444/base 2025-12-04T08:57:44.4732092Z * [new branch] gh/wconstab/444/head -> origin/gh/wconstab/444/head 2025-12-04T08:57:44.4733349Z * [new branch] gh/wconstab/444/orig -> origin/gh/wconstab/444/orig 2025-12-04T08:57:44.4734884Z * [new branch] gh/wconstab/447/base -> origin/gh/wconstab/447/base 2025-12-04T08:57:44.4736087Z * [new branch] gh/wconstab/447/head -> origin/gh/wconstab/447/head 2025-12-04T08:57:44.4737329Z * [new branch] gh/wconstab/447/orig -> origin/gh/wconstab/447/orig 2025-12-04T08:57:44.4738774Z * [new branch] gh/wconstab/448/base -> origin/gh/wconstab/448/base 2025-12-04T08:57:44.4739973Z * [new branch] gh/wconstab/448/head -> origin/gh/wconstab/448/head 2025-12-04T08:57:44.4741035Z * [new branch] gh/wconstab/448/orig -> origin/gh/wconstab/448/orig 2025-12-04T08:57:44.4742434Z * [new branch] gh/wconstab/449/base -> origin/gh/wconstab/449/base 2025-12-04T08:57:44.4743568Z * [new branch] gh/wconstab/449/head -> origin/gh/wconstab/449/head 2025-12-04T08:57:44.4744710Z * [new branch] gh/wconstab/449/orig -> origin/gh/wconstab/449/orig 2025-12-04T08:57:44.4746016Z * [new branch] gh/wconstab/450/base -> origin/gh/wconstab/450/base 2025-12-04T08:57:44.4747196Z * [new branch] gh/wconstab/450/head -> origin/gh/wconstab/450/head 2025-12-04T08:57:44.4748747Z * [new branch] gh/wconstab/450/orig -> origin/gh/wconstab/450/orig 2025-12-04T08:57:44.4750518Z * [new branch] gh/wconstab/451/base -> origin/gh/wconstab/451/base 2025-12-04T08:57:44.4751787Z * [new branch] gh/wconstab/451/head -> origin/gh/wconstab/451/head 2025-12-04T08:57:44.4753017Z * [new branch] gh/wconstab/451/orig -> origin/gh/wconstab/451/orig 2025-12-04T08:57:44.4754589Z * [new branch] gh/wconstab/452/base -> origin/gh/wconstab/452/base 2025-12-04T08:57:44.4755693Z * [new branch] gh/wconstab/452/head -> origin/gh/wconstab/452/head 2025-12-04T08:57:44.4756777Z * [new branch] gh/wconstab/452/orig -> origin/gh/wconstab/452/orig 2025-12-04T08:57:44.4758309Z * [new branch] gh/wconstab/453/base -> origin/gh/wconstab/453/base 2025-12-04T08:57:44.4759491Z * [new branch] gh/wconstab/453/head -> origin/gh/wconstab/453/head 2025-12-04T08:57:44.4760762Z * [new branch] gh/wconstab/453/orig -> origin/gh/wconstab/453/orig 2025-12-04T08:57:44.4762086Z * [new branch] gh/wconstab/454/base -> origin/gh/wconstab/454/base 2025-12-04T08:57:44.4763168Z * [new branch] gh/wconstab/454/head -> origin/gh/wconstab/454/head 2025-12-04T08:57:44.4764274Z * [new branch] gh/wconstab/454/orig -> origin/gh/wconstab/454/orig 2025-12-04T08:57:44.4765751Z * [new branch] gh/wconstab/455/base -> origin/gh/wconstab/455/base 2025-12-04T08:57:44.4766823Z * [new branch] gh/wconstab/455/head -> origin/gh/wconstab/455/head 2025-12-04T08:57:44.4767998Z * [new branch] gh/wconstab/455/orig -> origin/gh/wconstab/455/orig 2025-12-04T08:57:44.4769764Z * [new branch] gh/wconstab/456/base -> origin/gh/wconstab/456/base 2025-12-04T08:57:44.4771179Z * [new branch] gh/wconstab/456/head -> origin/gh/wconstab/456/head 2025-12-04T08:57:44.4772396Z * [new branch] gh/wconstab/456/orig -> origin/gh/wconstab/456/orig 2025-12-04T08:57:44.4773877Z * [new branch] gh/wconstab/457/base -> origin/gh/wconstab/457/base 2025-12-04T08:57:44.4774940Z * [new branch] gh/wconstab/457/head -> origin/gh/wconstab/457/head 2025-12-04T08:57:44.4776113Z * [new branch] gh/wconstab/457/orig -> origin/gh/wconstab/457/orig 2025-12-04T08:57:44.4777654Z * [new branch] gh/wconstab/458/base -> origin/gh/wconstab/458/base 2025-12-04T08:57:44.4778785Z * [new branch] gh/wconstab/458/head -> origin/gh/wconstab/458/head 2025-12-04T08:57:44.4779866Z * [new branch] gh/wconstab/458/orig -> origin/gh/wconstab/458/orig 2025-12-04T08:57:44.4781242Z * [new branch] gh/wconstab/459/base -> origin/gh/wconstab/459/base 2025-12-04T08:57:44.4782384Z * [new branch] gh/wconstab/459/head -> origin/gh/wconstab/459/head 2025-12-04T08:57:44.4783480Z * [new branch] gh/wconstab/459/orig -> origin/gh/wconstab/459/orig 2025-12-04T08:57:44.4785522Z * [new branch] gh/wconstab/460/base -> origin/gh/wconstab/460/base 2025-12-04T08:57:44.4786921Z * [new branch] gh/wconstab/460/head -> origin/gh/wconstab/460/head 2025-12-04T08:57:44.4788547Z * [new branch] gh/wconstab/460/orig -> origin/gh/wconstab/460/orig 2025-12-04T08:57:44.4790159Z * [new branch] gh/wconstab/461/base -> origin/gh/wconstab/461/base 2025-12-04T08:57:44.4791284Z * [new branch] gh/wconstab/461/head -> origin/gh/wconstab/461/head 2025-12-04T08:57:44.4792419Z * [new branch] gh/wconstab/461/orig -> origin/gh/wconstab/461/orig 2025-12-04T08:57:44.4793797Z * [new branch] gh/wconstab/462/base -> origin/gh/wconstab/462/base 2025-12-04T08:57:44.4795022Z * [new branch] gh/wconstab/462/head -> origin/gh/wconstab/462/head 2025-12-04T08:57:44.4796381Z * [new branch] gh/wconstab/462/orig -> origin/gh/wconstab/462/orig 2025-12-04T08:57:44.4797990Z * [new branch] gh/wconstab/463/base -> origin/gh/wconstab/463/base 2025-12-04T08:57:44.4799186Z * [new branch] gh/wconstab/463/head -> origin/gh/wconstab/463/head 2025-12-04T08:57:44.4800564Z * [new branch] gh/wconstab/463/orig -> origin/gh/wconstab/463/orig 2025-12-04T08:57:44.4802097Z * [new branch] gh/wconstab/464/base -> origin/gh/wconstab/464/base 2025-12-04T08:57:44.4803245Z * [new branch] gh/wconstab/464/head -> origin/gh/wconstab/464/head 2025-12-04T08:57:44.4804456Z * [new branch] gh/wconstab/464/orig -> origin/gh/wconstab/464/orig 2025-12-04T08:57:44.4805800Z * [new branch] gh/wconstab/465/base -> origin/gh/wconstab/465/base 2025-12-04T08:57:44.4806989Z * [new branch] gh/wconstab/465/head -> origin/gh/wconstab/465/head 2025-12-04T08:57:44.4808002Z * [new branch] gh/wconstab/465/orig -> origin/gh/wconstab/465/orig 2025-12-04T08:57:44.4809557Z * [new branch] gh/wconstab/466/base -> origin/gh/wconstab/466/base 2025-12-04T08:57:44.4810584Z * [new branch] gh/wconstab/466/head -> origin/gh/wconstab/466/head 2025-12-04T08:57:44.4811566Z * [new branch] gh/wconstab/466/orig -> origin/gh/wconstab/466/orig 2025-12-04T08:57:44.4813477Z * [new branch] gh/wconstab/467/base -> origin/gh/wconstab/467/base 2025-12-04T08:57:44.4814636Z * [new branch] gh/wconstab/467/head -> origin/gh/wconstab/467/head 2025-12-04T08:57:44.4816362Z * [new branch] gh/wconstab/467/orig -> origin/gh/wconstab/467/orig 2025-12-04T08:57:44.4817742Z * [new branch] gh/wconstab/468/base -> origin/gh/wconstab/468/base 2025-12-04T08:57:44.4819390Z * [new branch] gh/wconstab/468/head -> origin/gh/wconstab/468/head 2025-12-04T08:57:44.4820495Z * [new branch] gh/wconstab/468/orig -> origin/gh/wconstab/468/orig 2025-12-04T08:57:44.4822320Z * [new branch] gh/weifengpy/39/base -> origin/gh/weifengpy/39/base 2025-12-04T08:57:44.4823397Z * [new branch] gh/weifengpy/39/head -> origin/gh/weifengpy/39/head 2025-12-04T08:57:44.4824925Z * [new branch] gh/weifengpy/39/orig -> origin/gh/weifengpy/39/orig 2025-12-04T08:57:44.4826703Z * [new branch] gh/weifengpy/40/base -> origin/gh/weifengpy/40/base 2025-12-04T08:57:44.4828033Z * [new branch] gh/weifengpy/40/head -> origin/gh/weifengpy/40/head 2025-12-04T08:57:44.4829211Z * [new branch] gh/weifengpy/40/orig -> origin/gh/weifengpy/40/orig 2025-12-04T08:57:44.4830773Z * [new branch] gh/weifengpy/41/base -> origin/gh/weifengpy/41/base 2025-12-04T08:57:44.4832158Z * [new branch] gh/weifengpy/41/head -> origin/gh/weifengpy/41/head 2025-12-04T08:57:44.4833435Z * [new branch] gh/weifengpy/41/orig -> origin/gh/weifengpy/41/orig 2025-12-04T08:57:44.4835447Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-12-04T08:57:44.4836524Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-12-04T08:57:44.4837671Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-12-04T08:57:44.4839330Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-12-04T08:57:44.4840627Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-12-04T08:57:44.4841728Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-12-04T08:57:44.4843230Z * [new branch] gh/williamwen42/282/base -> origin/gh/williamwen42/282/base 2025-12-04T08:57:44.4844327Z * [new branch] gh/williamwen42/282/head -> origin/gh/williamwen42/282/head 2025-12-04T08:57:44.4845438Z * [new branch] gh/williamwen42/282/orig -> origin/gh/williamwen42/282/orig 2025-12-04T08:57:44.4847048Z * [new branch] gh/williamwen42/287/base -> origin/gh/williamwen42/287/base 2025-12-04T08:57:44.4848184Z * [new branch] gh/williamwen42/287/head -> origin/gh/williamwen42/287/head 2025-12-04T08:57:44.4849329Z * [new branch] gh/williamwen42/287/orig -> origin/gh/williamwen42/287/orig 2025-12-04T08:57:44.4850857Z * [new branch] gh/williamwen42/288/base -> origin/gh/williamwen42/288/base 2025-12-04T08:57:44.4852097Z * [new branch] gh/williamwen42/288/head -> origin/gh/williamwen42/288/head 2025-12-04T08:57:44.4853070Z * [new branch] gh/williamwen42/288/orig -> origin/gh/williamwen42/288/orig 2025-12-04T08:57:44.4855194Z * [new branch] gh/williamwen42/296/base -> origin/gh/williamwen42/296/base 2025-12-04T08:57:44.4856466Z * [new branch] gh/williamwen42/296/head -> origin/gh/williamwen42/296/head 2025-12-04T08:57:44.4857561Z * [new branch] gh/williamwen42/296/orig -> origin/gh/williamwen42/296/orig 2025-12-04T08:57:44.4858960Z * [new branch] gh/williamwen42/297/base -> origin/gh/williamwen42/297/base 2025-12-04T08:57:44.4860070Z * [new branch] gh/williamwen42/297/head -> origin/gh/williamwen42/297/head 2025-12-04T08:57:44.4861160Z * [new branch] gh/williamwen42/297/orig -> origin/gh/williamwen42/297/orig 2025-12-04T08:57:44.4862839Z * [new branch] gh/williamwen42/306/base -> origin/gh/williamwen42/306/base 2025-12-04T08:57:44.4864000Z * [new branch] gh/williamwen42/306/head -> origin/gh/williamwen42/306/head 2025-12-04T08:57:44.4865332Z * [new branch] gh/williamwen42/306/orig -> origin/gh/williamwen42/306/orig 2025-12-04T08:57:44.4866839Z * [new branch] gh/williamwen42/309/base -> origin/gh/williamwen42/309/base 2025-12-04T08:57:44.4868381Z * [new branch] gh/williamwen42/309/head -> origin/gh/williamwen42/309/head 2025-12-04T08:57:44.4869526Z * [new branch] gh/williamwen42/309/orig -> origin/gh/williamwen42/309/orig 2025-12-04T08:57:44.4871035Z * [new branch] gh/williamwen42/310/base -> origin/gh/williamwen42/310/base 2025-12-04T08:57:44.4872173Z * [new branch] gh/williamwen42/310/head -> origin/gh/williamwen42/310/head 2025-12-04T08:57:44.4873270Z * [new branch] gh/williamwen42/310/orig -> origin/gh/williamwen42/310/orig 2025-12-04T08:57:44.4876220Z * [new branch] gh/williamwen42/311/base -> origin/gh/williamwen42/311/base 2025-12-04T08:57:44.4877330Z * [new branch] gh/williamwen42/311/head -> origin/gh/williamwen42/311/head 2025-12-04T08:57:44.4878470Z * [new branch] gh/williamwen42/311/orig -> origin/gh/williamwen42/311/orig 2025-12-04T08:57:44.4880057Z * [new branch] gh/williamwen42/319/base -> origin/gh/williamwen42/319/base 2025-12-04T08:57:44.4881153Z * [new branch] gh/williamwen42/319/head -> origin/gh/williamwen42/319/head 2025-12-04T08:57:44.4882255Z * [new branch] gh/williamwen42/319/orig -> origin/gh/williamwen42/319/orig 2025-12-04T08:57:44.4883861Z * [new branch] gh/williamwen42/325/base -> origin/gh/williamwen42/325/base 2025-12-04T08:57:44.4885048Z * [new branch] gh/williamwen42/325/head -> origin/gh/williamwen42/325/head 2025-12-04T08:57:44.4886132Z * [new branch] gh/williamwen42/325/orig -> origin/gh/williamwen42/325/orig 2025-12-04T08:57:44.4887626Z * [new branch] gh/williamwen42/326/base -> origin/gh/williamwen42/326/base 2025-12-04T08:57:44.4888834Z * [new branch] gh/williamwen42/326/head -> origin/gh/williamwen42/326/head 2025-12-04T08:57:44.4889950Z * [new branch] gh/williamwen42/326/orig -> origin/gh/williamwen42/326/orig 2025-12-04T08:57:44.4891496Z * [new branch] gh/williamwen42/327/base -> origin/gh/williamwen42/327/base 2025-12-04T08:57:44.4894415Z * [new branch] gh/williamwen42/327/head -> origin/gh/williamwen42/327/head 2025-12-04T08:57:44.4895429Z * [new branch] gh/williamwen42/327/orig -> origin/gh/williamwen42/327/orig 2025-12-04T08:57:44.4896099Z * [new branch] gh/williamwen42/328/base -> origin/gh/williamwen42/328/base 2025-12-04T08:57:44.4896724Z * [new branch] gh/williamwen42/328/head -> origin/gh/williamwen42/328/head 2025-12-04T08:57:44.4897684Z * [new branch] gh/williamwen42/328/orig -> origin/gh/williamwen42/328/orig 2025-12-04T08:57:44.4899467Z * [new branch] gh/williamwen42/329/base -> origin/gh/williamwen42/329/base 2025-12-04T08:57:44.4900684Z * [new branch] gh/williamwen42/329/head -> origin/gh/williamwen42/329/head 2025-12-04T08:57:44.4901807Z * [new branch] gh/williamwen42/329/orig -> origin/gh/williamwen42/329/orig 2025-12-04T08:57:44.4903336Z * [new branch] gh/williamwen42/330/base -> origin/gh/williamwen42/330/base 2025-12-04T08:57:44.4904443Z * [new branch] gh/williamwen42/330/head -> origin/gh/williamwen42/330/head 2025-12-04T08:57:44.4905559Z * [new branch] gh/williamwen42/330/orig -> origin/gh/williamwen42/330/orig 2025-12-04T08:57:44.4907151Z * [new branch] gh/williamwen42/331/base -> origin/gh/williamwen42/331/base 2025-12-04T08:57:44.4908715Z * [new branch] gh/williamwen42/331/head -> origin/gh/williamwen42/331/head 2025-12-04T08:57:44.4909812Z * [new branch] gh/williamwen42/331/orig -> origin/gh/williamwen42/331/orig 2025-12-04T08:57:44.4911337Z * [new branch] gh/williamwen42/332/base -> origin/gh/williamwen42/332/base 2025-12-04T08:57:44.4912456Z * [new branch] gh/williamwen42/332/head -> origin/gh/williamwen42/332/head 2025-12-04T08:57:44.4913563Z * [new branch] gh/williamwen42/332/orig -> origin/gh/williamwen42/332/orig 2025-12-04T08:57:44.4915315Z * [new branch] gh/williamwen42/333/base -> origin/gh/williamwen42/333/base 2025-12-04T08:57:44.4916409Z * [new branch] gh/williamwen42/333/head -> origin/gh/williamwen42/333/head 2025-12-04T08:57:44.4917580Z * [new branch] gh/williamwen42/333/orig -> origin/gh/williamwen42/333/orig 2025-12-04T08:57:44.4919708Z * [new branch] gh/williamwen42/334/base -> origin/gh/williamwen42/334/base 2025-12-04T08:57:44.4920943Z * [new branch] gh/williamwen42/334/head -> origin/gh/williamwen42/334/head 2025-12-04T08:57:44.4922033Z * [new branch] gh/williamwen42/334/orig -> origin/gh/williamwen42/334/orig 2025-12-04T08:57:44.4923766Z * [new branch] gh/williamwen42/335/base -> origin/gh/williamwen42/335/base 2025-12-04T08:57:44.4930716Z * [new branch] gh/williamwen42/335/head -> origin/gh/williamwen42/335/head 2025-12-04T08:57:44.4931895Z * [new branch] gh/williamwen42/335/orig -> origin/gh/williamwen42/335/orig 2025-12-04T08:57:44.4933679Z * [new branch] gh/williamwen42/336/base -> origin/gh/williamwen42/336/base 2025-12-04T08:57:44.4934735Z * [new branch] gh/williamwen42/336/head -> origin/gh/williamwen42/336/head 2025-12-04T08:57:44.4935826Z * [new branch] gh/williamwen42/336/orig -> origin/gh/williamwen42/336/orig 2025-12-04T08:57:44.4937496Z * [new branch] gh/williamwen42/337/base -> origin/gh/williamwen42/337/base 2025-12-04T08:57:44.4938602Z * [new branch] gh/williamwen42/337/head -> origin/gh/williamwen42/337/head 2025-12-04T08:57:44.4939922Z * [new branch] gh/williamwen42/337/orig -> origin/gh/williamwen42/337/orig 2025-12-04T08:57:44.4941511Z * [new branch] gh/williamwen42/338/base -> origin/gh/williamwen42/338/base 2025-12-04T08:57:44.4942675Z * [new branch] gh/williamwen42/338/head -> origin/gh/williamwen42/338/head 2025-12-04T08:57:44.4943742Z * [new branch] gh/williamwen42/338/orig -> origin/gh/williamwen42/338/orig 2025-12-04T08:57:44.4945265Z * [new branch] gh/williamwen42/339/base -> origin/gh/williamwen42/339/base 2025-12-04T08:57:44.4946358Z * [new branch] gh/williamwen42/339/head -> origin/gh/williamwen42/339/head 2025-12-04T08:57:44.4947528Z * [new branch] gh/williamwen42/339/orig -> origin/gh/williamwen42/339/orig 2025-12-04T08:57:44.4949734Z * [new branch] gh/williamwen42/340/base -> origin/gh/williamwen42/340/base 2025-12-04T08:57:44.4950580Z * [new branch] gh/williamwen42/340/head -> origin/gh/williamwen42/340/head 2025-12-04T08:57:44.4951735Z * [new branch] gh/williamwen42/340/orig -> origin/gh/williamwen42/340/orig 2025-12-04T08:57:44.4953389Z * [new branch] gh/williamwen42/341/base -> origin/gh/williamwen42/341/base 2025-12-04T08:57:44.4954563Z * [new branch] gh/williamwen42/341/head -> origin/gh/williamwen42/341/head 2025-12-04T08:57:44.4955658Z * [new branch] gh/williamwen42/341/orig -> origin/gh/williamwen42/341/orig 2025-12-04T08:57:44.4957198Z * [new branch] gh/williamwen42/342/base -> origin/gh/williamwen42/342/base 2025-12-04T08:57:44.4958384Z * [new branch] gh/williamwen42/342/head -> origin/gh/williamwen42/342/head 2025-12-04T08:57:44.4959493Z * [new branch] gh/williamwen42/342/orig -> origin/gh/williamwen42/342/orig 2025-12-04T08:57:44.4961168Z * [new branch] gh/williamwen42/343/base -> origin/gh/williamwen42/343/base 2025-12-04T08:57:44.4962303Z * [new branch] gh/williamwen42/343/head -> origin/gh/williamwen42/343/head 2025-12-04T08:57:44.4963404Z * [new branch] gh/williamwen42/343/orig -> origin/gh/williamwen42/343/orig 2025-12-04T08:57:44.4965018Z * [new branch] gh/williamwen42/344/base -> origin/gh/williamwen42/344/base 2025-12-04T08:57:44.4966120Z * [new branch] gh/williamwen42/344/head -> origin/gh/williamwen42/344/head 2025-12-04T08:57:44.4967232Z * [new branch] gh/williamwen42/344/orig -> origin/gh/williamwen42/344/orig 2025-12-04T08:57:44.4968825Z * [new branch] gh/williamwen42/345/base -> origin/gh/williamwen42/345/base 2025-12-04T08:57:44.4969968Z * [new branch] gh/williamwen42/345/head -> origin/gh/williamwen42/345/head 2025-12-04T08:57:44.4971088Z * [new branch] gh/williamwen42/345/orig -> origin/gh/williamwen42/345/orig 2025-12-04T08:57:44.4972581Z * [new branch] gh/williamwen42/346/base -> origin/gh/williamwen42/346/base 2025-12-04T08:57:44.4973753Z * [new branch] gh/williamwen42/346/head -> origin/gh/williamwen42/346/head 2025-12-04T08:57:44.4974875Z * [new branch] gh/williamwen42/346/orig -> origin/gh/williamwen42/346/orig 2025-12-04T08:57:44.4976520Z * [new branch] gh/williamwen42/347/base -> origin/gh/williamwen42/347/base 2025-12-04T08:57:44.4977583Z * [new branch] gh/williamwen42/347/head -> origin/gh/williamwen42/347/head 2025-12-04T08:57:44.4978653Z * [new branch] gh/williamwen42/347/orig -> origin/gh/williamwen42/347/orig 2025-12-04T08:57:44.4980243Z * [new branch] gh/williamwen42/348/base -> origin/gh/williamwen42/348/base 2025-12-04T08:57:44.4981268Z * [new branch] gh/williamwen42/348/head -> origin/gh/williamwen42/348/head 2025-12-04T08:57:44.4982320Z * [new branch] gh/williamwen42/348/orig -> origin/gh/williamwen42/348/orig 2025-12-04T08:57:44.4983724Z * [new branch] gh/williamwen42/349/base -> origin/gh/williamwen42/349/base 2025-12-04T08:57:44.4984820Z * [new branch] gh/williamwen42/349/head -> origin/gh/williamwen42/349/head 2025-12-04T08:57:44.4985934Z * [new branch] gh/williamwen42/349/orig -> origin/gh/williamwen42/349/orig 2025-12-04T08:57:44.4987524Z * [new branch] gh/williamwen42/350/base -> origin/gh/williamwen42/350/base 2025-12-04T08:57:44.4988929Z * [new branch] gh/williamwen42/350/head -> origin/gh/williamwen42/350/head 2025-12-04T08:57:44.4990058Z * [new branch] gh/williamwen42/350/orig -> origin/gh/williamwen42/350/orig 2025-12-04T08:57:44.4991787Z * [new branch] gh/williamwen42/351/base -> origin/gh/williamwen42/351/base 2025-12-04T08:57:44.4992951Z * [new branch] gh/williamwen42/351/head -> origin/gh/williamwen42/351/head 2025-12-04T08:57:44.4994065Z * [new branch] gh/williamwen42/351/orig -> origin/gh/williamwen42/351/orig 2025-12-04T08:57:44.4995711Z * [new branch] gh/williamwen42/352/base -> origin/gh/williamwen42/352/base 2025-12-04T08:57:44.4997314Z * [new branch] gh/williamwen42/352/head -> origin/gh/williamwen42/352/head 2025-12-04T08:57:44.4998486Z * [new branch] gh/williamwen42/352/orig -> origin/gh/williamwen42/352/orig 2025-12-04T08:57:44.5000342Z * [new branch] gh/williamwen42/353/base -> origin/gh/williamwen42/353/base 2025-12-04T08:57:44.5001477Z * [new branch] gh/williamwen42/353/head -> origin/gh/williamwen42/353/head 2025-12-04T08:57:44.5002614Z * [new branch] gh/williamwen42/353/orig -> origin/gh/williamwen42/353/orig 2025-12-04T08:57:44.5004063Z * [new branch] gh/williamwen42/354/base -> origin/gh/williamwen42/354/base 2025-12-04T08:57:44.5005240Z * [new branch] gh/williamwen42/354/head -> origin/gh/williamwen42/354/head 2025-12-04T08:57:44.5006475Z * [new branch] gh/williamwen42/354/orig -> origin/gh/williamwen42/354/orig 2025-12-04T08:57:44.5008046Z * [new branch] gh/williamwen42/355/base -> origin/gh/williamwen42/355/base 2025-12-04T08:57:44.5009175Z * [new branch] gh/williamwen42/355/head -> origin/gh/williamwen42/355/head 2025-12-04T08:57:44.5010242Z * [new branch] gh/williamwen42/355/orig -> origin/gh/williamwen42/355/orig 2025-12-04T08:57:44.5011955Z * [new branch] gh/williamwen42/356/base -> origin/gh/williamwen42/356/base 2025-12-04T08:57:44.5013018Z * [new branch] gh/williamwen42/356/head -> origin/gh/williamwen42/356/head 2025-12-04T08:57:44.5014149Z * [new branch] gh/williamwen42/356/orig -> origin/gh/williamwen42/356/orig 2025-12-04T08:57:44.5015647Z * [new branch] gh/williamwen42/357/base -> origin/gh/williamwen42/357/base 2025-12-04T08:57:44.5016776Z * [new branch] gh/williamwen42/357/head -> origin/gh/williamwen42/357/head 2025-12-04T08:57:44.5017865Z * [new branch] gh/williamwen42/357/orig -> origin/gh/williamwen42/357/orig 2025-12-04T08:57:44.5019357Z * [new branch] gh/williamwen42/358/base -> origin/gh/williamwen42/358/base 2025-12-04T08:57:44.5020445Z * [new branch] gh/williamwen42/358/head -> origin/gh/williamwen42/358/head 2025-12-04T08:57:44.5021593Z * [new branch] gh/williamwen42/358/orig -> origin/gh/williamwen42/358/orig 2025-12-04T08:57:44.5023326Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-12-04T08:57:44.5024938Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-12-04T08:57:44.5026399Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-12-04T08:57:44.5027545Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-12-04T08:57:44.5029083Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-12-04T08:57:44.5030201Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-12-04T08:57:44.5031307Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-12-04T08:57:44.5032768Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-12-04T08:57:44.5033853Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-12-04T08:57:44.5034966Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-12-04T08:57:44.5036457Z * [new branch] gh/xmfan/301/base -> origin/gh/xmfan/301/base 2025-12-04T08:57:44.5037685Z * [new branch] gh/xmfan/301/head -> origin/gh/xmfan/301/head 2025-12-04T08:57:44.5038646Z * [new branch] gh/xmfan/301/orig -> origin/gh/xmfan/301/orig 2025-12-04T08:57:44.5040251Z * [new branch] gh/xmfan/304/base -> origin/gh/xmfan/304/base 2025-12-04T08:57:44.5041425Z * [new branch] gh/xmfan/304/head -> origin/gh/xmfan/304/head 2025-12-04T08:57:44.5042497Z * [new branch] gh/xmfan/304/orig -> origin/gh/xmfan/304/orig 2025-12-04T08:57:44.5043900Z * [new branch] gh/xmfan/309/base -> origin/gh/xmfan/309/base 2025-12-04T08:57:44.5045030Z * [new branch] gh/xmfan/309/head -> origin/gh/xmfan/309/head 2025-12-04T08:57:44.5046145Z * [new branch] gh/xmfan/309/orig -> origin/gh/xmfan/309/orig 2025-12-04T08:57:44.5047605Z * [new branch] gh/xmfan/310/base -> origin/gh/xmfan/310/base 2025-12-04T08:57:44.5048675Z * [new branch] gh/xmfan/310/head -> origin/gh/xmfan/310/head 2025-12-04T08:57:44.5049767Z * [new branch] gh/xmfan/310/orig -> origin/gh/xmfan/310/orig 2025-12-04T08:57:44.5051224Z * [new branch] gh/xmfan/311/base -> origin/gh/xmfan/311/base 2025-12-04T08:57:44.5052292Z * [new branch] gh/xmfan/311/head -> origin/gh/xmfan/311/head 2025-12-04T08:57:44.5053399Z * [new branch] gh/xmfan/311/orig -> origin/gh/xmfan/311/orig 2025-12-04T08:57:44.5054817Z * [new branch] gh/xmfan/312/base -> origin/gh/xmfan/312/base 2025-12-04T08:57:44.5056058Z * [new branch] gh/xmfan/312/head -> origin/gh/xmfan/312/head 2025-12-04T08:57:44.5057110Z * [new branch] gh/xmfan/312/orig -> origin/gh/xmfan/312/orig 2025-12-04T08:57:44.5058533Z * [new branch] gh/xmfan/313/base -> origin/gh/xmfan/313/base 2025-12-04T08:57:44.5059602Z * [new branch] gh/xmfan/313/head -> origin/gh/xmfan/313/head 2025-12-04T08:57:44.5060658Z * [new branch] gh/xmfan/313/orig -> origin/gh/xmfan/313/orig 2025-12-04T08:57:44.5062476Z * [new branch] gh/xuanzhang816/27/base -> origin/gh/xuanzhang816/27/base 2025-12-04T08:57:44.5063551Z * [new branch] gh/xuanzhang816/27/head -> origin/gh/xuanzhang816/27/head 2025-12-04T08:57:44.5064651Z * [new branch] gh/xuanzhang816/27/orig -> origin/gh/xuanzhang816/27/orig 2025-12-04T08:57:44.5066194Z * [new branch] gh/xuanzhang816/32/base -> origin/gh/xuanzhang816/32/base 2025-12-04T08:57:44.5067315Z * [new branch] gh/xuanzhang816/32/head -> origin/gh/xuanzhang816/32/head 2025-12-04T08:57:44.5068727Z * [new branch] gh/xuanzhang816/32/orig -> origin/gh/xuanzhang816/32/orig 2025-12-04T08:57:44.5070293Z * [new branch] gh/xuanzhang816/33/base -> origin/gh/xuanzhang816/33/base 2025-12-04T08:57:44.5071605Z * [new branch] gh/xuanzhang816/33/head -> origin/gh/xuanzhang816/33/head 2025-12-04T08:57:44.5072702Z * [new branch] gh/xuanzhang816/33/orig -> origin/gh/xuanzhang816/33/orig 2025-12-04T08:57:44.5074643Z * [new branch] gh/xuanzhang816/34/base -> origin/gh/xuanzhang816/34/base 2025-12-04T08:57:44.5075753Z * [new branch] gh/xuanzhang816/34/head -> origin/gh/xuanzhang816/34/head 2025-12-04T08:57:44.5076866Z * [new branch] gh/xuanzhang816/34/orig -> origin/gh/xuanzhang816/34/orig 2025-12-04T08:57:44.5078629Z * [new branch] gh/xuanzhang816/35/base -> origin/gh/xuanzhang816/35/base 2025-12-04T08:57:44.5079851Z * [new branch] gh/xuanzhang816/35/head -> origin/gh/xuanzhang816/35/head 2025-12-04T08:57:44.5080936Z * [new branch] gh/xuanzhang816/35/orig -> origin/gh/xuanzhang816/35/orig 2025-12-04T08:57:44.5082826Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-12-04T08:57:44.5083916Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-12-04T08:57:44.5085008Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-12-04T08:57:44.5086536Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-12-04T08:57:44.5087613Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-12-04T08:57:44.5088706Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-12-04T08:57:44.5090226Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-12-04T08:57:44.5091733Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-12-04T08:57:44.5092819Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-12-04T08:57:44.5094244Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-12-04T08:57:44.5095330Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-12-04T08:57:44.5096468Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-12-04T08:57:44.5097845Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-12-04T08:57:44.5098964Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-12-04T08:57:44.5100051Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-12-04T08:57:44.5101526Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-12-04T08:57:44.5102574Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-12-04T08:57:44.5103655Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-12-04T08:57:44.5105117Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-12-04T08:57:44.5106227Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-12-04T08:57:44.5107359Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-12-04T08:57:44.5109096Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-12-04T08:57:44.5110209Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-12-04T08:57:44.5111354Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-12-04T08:57:44.5112946Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-12-04T08:57:44.5114113Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-12-04T08:57:44.5115711Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-12-04T08:57:44.5116848Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-12-04T08:57:44.5117957Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-12-04T08:57:44.5120090Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-12-04T08:57:44.5121181Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-12-04T08:57:44.5122258Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-12-04T08:57:44.5123826Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-12-04T08:57:44.5125293Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-12-04T08:57:44.5126390Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-12-04T08:57:44.5127996Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-12-04T08:57:44.5128944Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-12-04T08:57:44.5130577Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-12-04T08:57:44.5132169Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-12-04T08:57:44.5133288Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-12-04T08:57:44.5134394Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-12-04T08:57:44.5136510Z * [new branch] gh/yang-yu-hang/1/base -> origin/gh/yang-yu-hang/1/base 2025-12-04T08:57:44.5137884Z * [new branch] gh/yang-yu-hang/1/head -> origin/gh/yang-yu-hang/1/head 2025-12-04T08:57:44.5139171Z * [new branch] gh/yang-yu-hang/1/orig -> origin/gh/yang-yu-hang/1/orig 2025-12-04T08:57:44.5140657Z * [new branch] gh/yang-yu-hang/2/base -> origin/gh/yang-yu-hang/2/base 2025-12-04T08:57:44.5142052Z * [new branch] gh/yang-yu-hang/2/head -> origin/gh/yang-yu-hang/2/head 2025-12-04T08:57:44.5143454Z * [new branch] gh/yang-yu-hang/2/orig -> origin/gh/yang-yu-hang/2/orig 2025-12-04T08:57:44.5144919Z * [new branch] gh/yang-yu-hang/3/base -> origin/gh/yang-yu-hang/3/base 2025-12-04T08:57:44.5146068Z * [new branch] gh/yang-yu-hang/3/head -> origin/gh/yang-yu-hang/3/head 2025-12-04T08:57:44.5147321Z * [new branch] gh/yang-yu-hang/3/orig -> origin/gh/yang-yu-hang/3/orig 2025-12-04T08:57:44.5149409Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-12-04T08:57:44.5150554Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-12-04T08:57:44.5151715Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-12-04T08:57:44.5153263Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-12-04T08:57:44.5154392Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-12-04T08:57:44.5155504Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-12-04T08:57:44.5156933Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-12-04T08:57:44.5158147Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-12-04T08:57:44.5159273Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-12-04T08:57:44.5160836Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-12-04T08:57:44.5161922Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-12-04T08:57:44.5163133Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-12-04T08:57:44.5164533Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-12-04T08:57:44.5165602Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-12-04T08:57:44.5166692Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-12-04T08:57:44.5168115Z * [new branch] gh/yangw-dev/26/base -> origin/gh/yangw-dev/26/base 2025-12-04T08:57:44.5169276Z * [new branch] gh/yangw-dev/26/head -> origin/gh/yangw-dev/26/head 2025-12-04T08:57:44.5170397Z * [new branch] gh/yangw-dev/26/orig -> origin/gh/yangw-dev/26/orig 2025-12-04T08:57:44.5171793Z * [new branch] gh/yangw-dev/27/base -> origin/gh/yangw-dev/27/base 2025-12-04T08:57:44.5172875Z * [new branch] gh/yangw-dev/27/head -> origin/gh/yangw-dev/27/head 2025-12-04T08:57:44.5173995Z * [new branch] gh/yangw-dev/27/orig -> origin/gh/yangw-dev/27/orig 2025-12-04T08:57:44.5175748Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-12-04T08:57:44.5176958Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-12-04T08:57:44.5178047Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-12-04T08:57:44.5179607Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-12-04T08:57:44.5180711Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-12-04T08:57:44.5181820Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-12-04T08:57:44.5183430Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-12-04T08:57:44.5184573Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-12-04T08:57:44.5185726Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-12-04T08:57:44.5187817Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-12-04T08:57:44.5189023Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-12-04T08:57:44.5190148Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-12-04T08:57:44.5191725Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-12-04T08:57:44.5193041Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-12-04T08:57:44.5194193Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-12-04T08:57:44.5195746Z * [new branch] gh/ydwu4/312/base -> origin/gh/ydwu4/312/base 2025-12-04T08:57:44.5196840Z * [new branch] gh/ydwu4/312/head -> origin/gh/ydwu4/312/head 2025-12-04T08:57:44.5197970Z * [new branch] gh/ydwu4/312/orig -> origin/gh/ydwu4/312/orig 2025-12-04T08:57:44.5199395Z * [new branch] gh/ydwu4/322/base -> origin/gh/ydwu4/322/base 2025-12-04T08:57:44.5200602Z * [new branch] gh/ydwu4/322/head -> origin/gh/ydwu4/322/head 2025-12-04T08:57:44.5201729Z * [new branch] gh/ydwu4/322/orig -> origin/gh/ydwu4/322/orig 2025-12-04T08:57:44.5203141Z * [new branch] gh/ydwu4/327/base -> origin/gh/ydwu4/327/base 2025-12-04T08:57:44.5204392Z * [new branch] gh/ydwu4/327/head -> origin/gh/ydwu4/327/head 2025-12-04T08:57:44.5205469Z * [new branch] gh/ydwu4/327/orig -> origin/gh/ydwu4/327/orig 2025-12-04T08:57:44.5207083Z * [new branch] gh/ydwu4/328/base -> origin/gh/ydwu4/328/base 2025-12-04T08:57:44.5208221Z * [new branch] gh/ydwu4/328/head -> origin/gh/ydwu4/328/head 2025-12-04T08:57:44.5209263Z * [new branch] gh/ydwu4/328/orig -> origin/gh/ydwu4/328/orig 2025-12-04T08:57:44.5210592Z * [new branch] gh/ydwu4/329/base -> origin/gh/ydwu4/329/base 2025-12-04T08:57:44.5211637Z * [new branch] gh/ydwu4/329/head -> origin/gh/ydwu4/329/head 2025-12-04T08:57:44.5212738Z * [new branch] gh/ydwu4/329/orig -> origin/gh/ydwu4/329/orig 2025-12-04T08:57:44.5214281Z * [new branch] gh/ydwu4/330/base -> origin/gh/ydwu4/330/base 2025-12-04T08:57:44.5215443Z * [new branch] gh/ydwu4/330/head -> origin/gh/ydwu4/330/head 2025-12-04T08:57:44.5216482Z * [new branch] gh/ydwu4/330/orig -> origin/gh/ydwu4/330/orig 2025-12-04T08:57:44.5217828Z * [new branch] gh/ydwu4/331/base -> origin/gh/ydwu4/331/base 2025-12-04T08:57:44.5218897Z * [new branch] gh/ydwu4/331/head -> origin/gh/ydwu4/331/head 2025-12-04T08:57:44.5220043Z * [new branch] gh/ydwu4/331/orig -> origin/gh/ydwu4/331/orig 2025-12-04T08:57:44.5221292Z * [new branch] gh/ydwu4/332/base -> origin/gh/ydwu4/332/base 2025-12-04T08:57:44.5222456Z * [new branch] gh/ydwu4/332/head -> origin/gh/ydwu4/332/head 2025-12-04T08:57:44.5223520Z * [new branch] gh/ydwu4/332/orig -> origin/gh/ydwu4/332/orig 2025-12-04T08:57:44.5225299Z * [new branch] gh/ydwu4/333/base -> origin/gh/ydwu4/333/base 2025-12-04T08:57:44.5226392Z * [new branch] gh/ydwu4/333/head -> origin/gh/ydwu4/333/head 2025-12-04T08:57:44.5227582Z * [new branch] gh/ydwu4/333/orig -> origin/gh/ydwu4/333/orig 2025-12-04T08:57:44.5229093Z * [new branch] gh/ydwu4/334/base -> origin/gh/ydwu4/334/base 2025-12-04T08:57:44.5230222Z * [new branch] gh/ydwu4/334/head -> origin/gh/ydwu4/334/head 2025-12-04T08:57:44.5231397Z * [new branch] gh/ydwu4/334/orig -> origin/gh/ydwu4/334/orig 2025-12-04T08:57:44.5232723Z * [new branch] gh/ydwu4/335/base -> origin/gh/ydwu4/335/base 2025-12-04T08:57:44.5233863Z * [new branch] gh/ydwu4/335/head -> origin/gh/ydwu4/335/head 2025-12-04T08:57:44.5234962Z * [new branch] gh/ydwu4/335/orig -> origin/gh/ydwu4/335/orig 2025-12-04T08:57:44.5236851Z * [new branch] gh/ydwu4/337/base -> origin/gh/ydwu4/337/base 2025-12-04T08:57:44.5238069Z * [new branch] gh/ydwu4/337/head -> origin/gh/ydwu4/337/head 2025-12-04T08:57:44.5239333Z * [new branch] gh/ydwu4/337/orig -> origin/gh/ydwu4/337/orig 2025-12-04T08:57:44.5240791Z * [new branch] gh/ydwu4/339/base -> origin/gh/ydwu4/339/base 2025-12-04T08:57:44.5241904Z * [new branch] gh/ydwu4/339/head -> origin/gh/ydwu4/339/head 2025-12-04T08:57:44.5242993Z * [new branch] gh/ydwu4/339/orig -> origin/gh/ydwu4/339/orig 2025-12-04T08:57:44.5244856Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-12-04T08:57:44.5245967Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-12-04T08:57:44.5247931Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-12-04T08:57:44.5249023Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-12-04T08:57:44.5251382Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-12-04T08:57:44.5252900Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-12-04T08:57:44.5254050Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-12-04T08:57:44.5255469Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-12-04T08:57:44.5256637Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-12-04T08:57:44.5257766Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-12-04T08:57:44.5259571Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-12-04T08:57:44.5260646Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-12-04T08:57:44.5261949Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-12-04T08:57:44.5263004Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-12-04T08:57:44.5264763Z * [new branch] gh/yushangdi/1/base -> origin/gh/yushangdi/1/base 2025-12-04T08:57:44.5265989Z * [new branch] gh/yushangdi/1/head -> origin/gh/yushangdi/1/head 2025-12-04T08:57:44.5267917Z * [new branch] gh/yushangdi/10/base -> origin/gh/yushangdi/10/base 2025-12-04T08:57:44.5269017Z * [new branch] gh/yushangdi/10/head -> origin/gh/yushangdi/10/head 2025-12-04T08:57:44.5270151Z * [new branch] gh/yushangdi/10/orig -> origin/gh/yushangdi/10/orig 2025-12-04T08:57:44.5271692Z * [new branch] gh/yushangdi/11/base -> origin/gh/yushangdi/11/base 2025-12-04T08:57:44.5272846Z * [new branch] gh/yushangdi/11/head -> origin/gh/yushangdi/11/head 2025-12-04T08:57:44.5274022Z * [new branch] gh/yushangdi/11/orig -> origin/gh/yushangdi/11/orig 2025-12-04T08:57:44.5275436Z * [new branch] gh/yushangdi/2/base -> origin/gh/yushangdi/2/base 2025-12-04T08:57:44.5276522Z * [new branch] gh/yushangdi/2/head -> origin/gh/yushangdi/2/head 2025-12-04T08:57:44.5278018Z * [new branch] gh/yushangdi/7/base -> origin/gh/yushangdi/7/base 2025-12-04T08:57:44.5279125Z * [new branch] gh/yushangdi/7/head -> origin/gh/yushangdi/7/head 2025-12-04T08:57:44.5280341Z * [new branch] gh/yushangdi/7/orig -> origin/gh/yushangdi/7/orig 2025-12-04T08:57:44.5282210Z * [new branch] gh/yushangdi/8/base -> origin/gh/yushangdi/8/base 2025-12-04T08:57:44.5283481Z * [new branch] gh/yushangdi/8/head -> origin/gh/yushangdi/8/head 2025-12-04T08:57:44.5284582Z * [new branch] gh/yushangdi/8/orig -> origin/gh/yushangdi/8/orig 2025-12-04T08:57:44.5285904Z * [new branch] gh/yushangdi/9/base -> origin/gh/yushangdi/9/base 2025-12-04T08:57:44.5287018Z * [new branch] gh/yushangdi/9/head -> origin/gh/yushangdi/9/head 2025-12-04T08:57:44.5288192Z * [new branch] gh/yushangdi/9/orig -> origin/gh/yushangdi/9/orig 2025-12-04T08:57:44.5289933Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-12-04T08:57:44.5291083Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-12-04T08:57:44.5292128Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-12-04T08:57:44.5293625Z * [new branch] gh/zklaus/20/base -> origin/gh/zklaus/20/base 2025-12-04T08:57:44.5294698Z * [new branch] gh/zklaus/20/head -> origin/gh/zklaus/20/head 2025-12-04T08:57:44.5295930Z * [new branch] gh/zklaus/20/orig -> origin/gh/zklaus/20/orig 2025-12-04T08:57:44.5297908Z * [new branch] gh/zklaus/21/base -> origin/gh/zklaus/21/base 2025-12-04T08:57:44.5299026Z * [new branch] gh/zklaus/21/head -> origin/gh/zklaus/21/head 2025-12-04T08:57:44.5300145Z * [new branch] gh/zklaus/21/orig -> origin/gh/zklaus/21/orig 2025-12-04T08:57:44.5302000Z * [new branch] gh/zklaus/22/base -> origin/gh/zklaus/22/base 2025-12-04T08:57:44.5303110Z * [new branch] gh/zklaus/22/head -> origin/gh/zklaus/22/head 2025-12-04T08:57:44.5304188Z * [new branch] gh/zklaus/22/orig -> origin/gh/zklaus/22/orig 2025-12-04T08:57:44.5305572Z * [new branch] gh/zklaus/23/base -> origin/gh/zklaus/23/base 2025-12-04T08:57:44.5306716Z * [new branch] gh/zklaus/23/head -> origin/gh/zklaus/23/head 2025-12-04T08:57:44.5308100Z * [new branch] gh/zklaus/23/orig -> origin/gh/zklaus/23/orig 2025-12-04T08:57:44.5309582Z * [new branch] gh/zklaus/24/base -> origin/gh/zklaus/24/base 2025-12-04T08:57:44.5310673Z * [new branch] gh/zklaus/24/head -> origin/gh/zklaus/24/head 2025-12-04T08:57:44.5311924Z * [new branch] gh/zklaus/24/orig -> origin/gh/zklaus/24/orig 2025-12-04T08:57:44.5313904Z * [new branch] gh/zou3519/1197/base -> origin/gh/zou3519/1197/base 2025-12-04T08:57:44.5315039Z * [new branch] gh/zou3519/1197/head -> origin/gh/zou3519/1197/head 2025-12-04T08:57:44.5316133Z * [new branch] gh/zou3519/1197/orig -> origin/gh/zou3519/1197/orig 2025-12-04T08:57:44.5317966Z * [new branch] gh/zou3519/1199/base -> origin/gh/zou3519/1199/base 2025-12-04T08:57:44.5319182Z * [new branch] gh/zou3519/1199/head -> origin/gh/zou3519/1199/head 2025-12-04T08:57:44.5320445Z * [new branch] gh/zou3519/1199/orig -> origin/gh/zou3519/1199/orig 2025-12-04T08:57:44.5321886Z * [new branch] gh/zou3519/1200/base -> origin/gh/zou3519/1200/base 2025-12-04T08:57:44.5323017Z * [new branch] gh/zou3519/1200/head -> origin/gh/zou3519/1200/head 2025-12-04T08:57:44.5326139Z * [new branch] gh/zou3519/1200/orig -> origin/gh/zou3519/1200/orig 2025-12-04T08:57:44.5327854Z * [new branch] gh/zou3519/1201/base -> origin/gh/zou3519/1201/base 2025-12-04T08:57:44.5329075Z * [new branch] gh/zou3519/1201/head -> origin/gh/zou3519/1201/head 2025-12-04T08:57:44.5330179Z * [new branch] gh/zou3519/1201/orig -> origin/gh/zou3519/1201/orig 2025-12-04T08:57:44.5331540Z * [new branch] gh/zou3519/1202/base -> origin/gh/zou3519/1202/base 2025-12-04T08:57:44.5332695Z * [new branch] gh/zou3519/1202/head -> origin/gh/zou3519/1202/head 2025-12-04T08:57:44.5333824Z * [new branch] gh/zou3519/1202/orig -> origin/gh/zou3519/1202/orig 2025-12-04T08:57:44.5335668Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-12-04T08:57:44.5336881Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-12-04T08:57:44.5338418Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-12-04T08:57:44.5339555Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-12-04T08:57:44.5340838Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-12-04T08:57:44.5342627Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-12-04T08:57:44.5343825Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-12-04T08:57:44.5345004Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-12-04T08:57:44.5346581Z * [new branch] gh/zpcore/13/base -> origin/gh/zpcore/13/base 2025-12-04T08:57:44.5347704Z * [new branch] gh/zpcore/13/head -> origin/gh/zpcore/13/head 2025-12-04T08:57:44.5348996Z * [new branch] gh/zpcore/13/orig -> origin/gh/zpcore/13/orig 2025-12-04T08:57:44.5351029Z * [new branch] gh/zpcore/14/base -> origin/gh/zpcore/14/base 2025-12-04T08:57:44.5352178Z * [new branch] gh/zpcore/14/head -> origin/gh/zpcore/14/head 2025-12-04T08:57:44.5353301Z * [new branch] gh/zpcore/14/orig -> origin/gh/zpcore/14/orig 2025-12-04T08:57:44.5355074Z * [new branch] gh/zpcore/15/base -> origin/gh/zpcore/15/base 2025-12-04T08:57:44.5356223Z * [new branch] gh/zpcore/15/head -> origin/gh/zpcore/15/head 2025-12-04T08:57:44.5357366Z * [new branch] gh/zpcore/15/orig -> origin/gh/zpcore/15/orig 2025-12-04T08:57:44.5358935Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-12-04T08:57:44.5360327Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-12-04T08:57:44.5362435Z * [new branch] gh/zpcore/21/base -> origin/gh/zpcore/21/base 2025-12-04T08:57:44.5364057Z * [new branch] gh/zpcore/21/head -> origin/gh/zpcore/21/head 2025-12-04T08:57:44.5365194Z * [new branch] gh/zpcore/21/orig -> origin/gh/zpcore/21/orig 2025-12-04T08:57:44.5367024Z * [new branch] gh/zpcore/22/base -> origin/gh/zpcore/22/base 2025-12-04T08:57:44.5368036Z * [new branch] gh/zpcore/22/head -> origin/gh/zpcore/22/head 2025-12-04T08:57:44.5369717Z * [new branch] gh/zpcore/22/orig -> origin/gh/zpcore/22/orig 2025-12-04T08:57:44.5371264Z * [new branch] gh/zpcore/23/base -> origin/gh/zpcore/23/base 2025-12-04T08:57:44.5372365Z * [new branch] gh/zpcore/23/head -> origin/gh/zpcore/23/head 2025-12-04T08:57:44.5373537Z * [new branch] gh/zpcore/23/orig -> origin/gh/zpcore/23/orig 2025-12-04T08:57:44.5374800Z * [new branch] gh/zpcore/24/base -> origin/gh/zpcore/24/base 2025-12-04T08:57:44.5375924Z * [new branch] gh/zpcore/24/head -> origin/gh/zpcore/24/head 2025-12-04T08:57:44.5377081Z * [new branch] gh/zpcore/24/orig -> origin/gh/zpcore/24/orig 2025-12-04T08:57:44.5378860Z * [new branch] gh/zpcore/25/base -> origin/gh/zpcore/25/base 2025-12-04T08:57:44.5379908Z * [new branch] gh/zpcore/25/head -> origin/gh/zpcore/25/head 2025-12-04T08:57:44.5381021Z * [new branch] gh/zpcore/25/orig -> origin/gh/zpcore/25/orig 2025-12-04T08:57:44.5382546Z * [new branch] gh/zpcore/26/base -> origin/gh/zpcore/26/base 2025-12-04T08:57:44.5383720Z * [new branch] gh/zpcore/26/head -> origin/gh/zpcore/26/head 2025-12-04T08:57:44.5384784Z * [new branch] gh/zpcore/26/orig -> origin/gh/zpcore/26/orig 2025-12-04T08:57:44.5386381Z * [new branch] gh/zpcore/27/base -> origin/gh/zpcore/27/base 2025-12-04T08:57:44.5387531Z * [new branch] gh/zpcore/27/head -> origin/gh/zpcore/27/head 2025-12-04T08:57:44.5388918Z * [new branch] gh/zpcore/27/orig -> origin/gh/zpcore/27/orig 2025-12-04T08:57:44.5390971Z * [new branch] gh/zpcore/28/base -> origin/gh/zpcore/28/base 2025-12-04T08:57:44.5392586Z * [new branch] gh/zpcore/28/head -> origin/gh/zpcore/28/head 2025-12-04T08:57:44.5393859Z * [new branch] gh/zpcore/28/orig -> origin/gh/zpcore/28/orig 2025-12-04T08:57:44.5395242Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-12-04T08:57:44.5396286Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-12-04T08:57:44.5397659Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-12-04T08:57:44.5398710Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-12-04T08:57:44.5400159Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-12-04T08:57:44.5401198Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-12-04T08:57:44.5402906Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-12-04T08:57:44.5403974Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-12-04T08:57:44.5405674Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-12-04T08:57:44.5406705Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-12-04T08:57:44.5408152Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-12-04T08:57:44.5409260Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-12-04T08:57:44.5410755Z * [new branch] google-main -> origin/google-main 2025-12-04T08:57:44.5412248Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-12-04T08:57:44.5413258Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-12-04T08:57:44.5415049Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-12-04T08:57:44.5416472Z * [new branch] hameerabbasi/complex_tensor_subclass -> origin/hameerabbasi/complex_tensor_subclass 2025-12-04T08:57:44.5417685Z * [new branch] hameerabbasi/fix-ctensor-gradcheck-tests -> origin/hameerabbasi/fix-ctensor-gradcheck-tests 2025-12-04T08:57:44.5418626Z * [new branch] hameerabbasi/gradcheck-allclose -> origin/hameerabbasi/gradcheck-allclose 2025-12-04T08:57:44.5419654Z * [new branch] hc_baseline -> origin/hc_baseline 2025-12-04T08:57:44.5421227Z * [new branch] hhh_rand -> origin/hhh_rand 2025-12-04T08:57:44.5422771Z * [new branch] huba/f1 -> origin/huba/f1 2025-12-04T08:57:44.5425065Z * [new branch] increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test -> origin/increase-timeout-linux-jammy-cuda12_8-py3_10-gcc11-test 2025-12-04T08:57:44.5425787Z * [new branch] inlining -> origin/inlining 2025-12-04T08:57:44.5427021Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-12-04T08:57:44.5428411Z * [new branch] install-torchao-0.13.0 -> origin/install-torchao-0.13.0 2025-12-04T08:57:44.5429917Z * [new branch] instrument-trunk-pull-linux-with-job-test-filters -> origin/instrument-trunk-pull-linux-with-job-test-filters 2025-12-04T08:57:44.5430717Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-12-04T08:57:44.5431962Z * [new branch] issue#58739 -> origin/issue#58739 2025-12-04T08:57:44.5433293Z * [new branch] jainapurva-patch-1 -> origin/jainapurva-patch-1 2025-12-04T08:57:44.5434689Z * [new branch] jathu/o3 -> origin/jathu/o3 2025-12-04T08:57:44.5435834Z * [new branch] jathu/sve -> origin/jathu/sve 2025-12-04T08:57:44.5437501Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-12-04T08:57:44.5438480Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-12-04T08:57:44.5440259Z * [new branch] jiannanWang/memorysnapshot_filter -> origin/jiannanWang/memorysnapshot_filter 2025-12-04T08:57:44.5441426Z * [new branch] jiannanWang/profilerstepwarning -> origin/jiannanWang/profilerstepwarning 2025-12-04T08:57:44.5442598Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-12-04T08:57:44.5443800Z * [new branch] jithunnair-amd-patch-10 -> origin/jithunnair-amd-patch-10 2025-12-04T08:57:44.5444983Z * [new branch] jithunnair-amd-patch-2 -> origin/jithunnair-amd-patch-2 2025-12-04T08:57:44.5446203Z * [new branch] jithunnair-amd-patch-3 -> origin/jithunnair-amd-patch-3 2025-12-04T08:57:44.5447366Z * [new branch] jithunnair-amd-patch-4 -> origin/jithunnair-amd-patch-4 2025-12-04T08:57:44.5448454Z * [new branch] jithunnair-amd-patch-5 -> origin/jithunnair-amd-patch-5 2025-12-04T08:57:44.5449635Z * [new branch] jithunnair-amd-patch-6 -> origin/jithunnair-amd-patch-6 2025-12-04T08:57:44.5450725Z * [new branch] jithunnair-amd-patch-7 -> origin/jithunnair-amd-patch-7 2025-12-04T08:57:44.5452044Z * [new branch] jithunnair-amd-patch-8 -> origin/jithunnair-amd-patch-8 2025-12-04T08:57:44.5453166Z * [new branch] jithunnair-amd-patch-9 -> origin/jithunnair-amd-patch-9 2025-12-04T08:57:44.5454715Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-12-04T08:57:44.5456165Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-12-04T08:57:44.5457474Z * [new branch] kainan_test -> origin/kainan_test 2025-12-04T08:57:44.5458558Z * [new branch] larryliu0820-patch-1 -> origin/larryliu0820-patch-1 2025-12-04T08:57:44.5460072Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-12-04T08:57:44.5461617Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-12-04T08:57:44.5462943Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-12-04T08:57:44.5464049Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-12-04T08:57:44.5465012Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-12-04T08:57:44.5466029Z * [new branch] llama4-stable -> origin/llama4-stable 2025-12-04T08:57:44.5468195Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-12-04T08:57:44.5469768Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-12-04T08:57:44.5470965Z * [new branch] lucaskabela/fix_164876 -> origin/lucaskabela/fix_164876 2025-12-04T08:57:44.5472027Z * [new branch] lucaskabela/flop_counter -> origin/lucaskabela/flop_counter 2025-12-04T08:57:44.5473139Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-12-04T08:57:44.5474239Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-12-04T08:57:44.5475549Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-12-04T08:57:44.5476946Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-12-04T08:57:44.5478560Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-12-04T08:57:44.5479481Z * [new branch] lucaskabela/rnn_decomp -> origin/lucaskabela/rnn_decomp 2025-12-04T08:57:44.5480781Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-12-04T08:57:44.5481870Z * [new branch] lucaskabela/typing_ctx_manager -> origin/lucaskabela/typing_ctx_manager 2025-12-04T08:57:44.5482968Z * [new branch] lucaskabela/typing_nn_module -> origin/lucaskabela/typing_nn_module 2025-12-04T08:57:44.5484067Z * [new branch] lucaskabela/typing_user_defined -> origin/lucaskabela/typing_user_defined 2025-12-04T08:57:44.5485145Z * [new branch] lucaskabela/typing_variables -> origin/lucaskabela/typing_variables 2025-12-04T08:57:44.5486252Z * [new branch] lucaskabela/typing_variables_dicts -> origin/lucaskabela/typing_variables_dicts 2025-12-04T08:57:44.5487408Z * [new branch] lucaskabela/typing_variables_functions -> origin/lucaskabela/typing_variables_functions 2025-12-04T08:57:44.5488379Z * [new branch] lucaskabela/typing_variables_lists -> origin/lucaskabela/typing_variables_lists 2025-12-04T08:57:44.5490102Z * [new branch] lw/torch_box_by_ref -> origin/lw/torch_box_by_ref 2025-12-04T08:57:44.5491271Z * [new branch] main -> origin/main 2025-12-04T08:57:44.5492599Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-12-04T08:57:44.5493892Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-12-04T08:57:44.5495086Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-12-04T08:57:44.5496308Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-12-04T08:57:44.5497537Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-12-04T08:57:44.5498715Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-12-04T08:57:44.5499864Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-12-04T08:57:44.5501049Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-12-04T08:57:44.5502637Z * [new branch] malfet/add-3.14-ci -> origin/malfet/add-3.14-ci 2025-12-04T08:57:44.5503979Z * [new branch] malfet/be-do-not-make-typos-in-build-artifacts -> origin/malfet/be-do-not-make-typos-in-build-artifacts 2025-12-04T08:57:44.5505095Z * [new branch] malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch 2025-12-04T08:57:44.5506322Z * [new branch] malfet/be-remove-misisng-neon-headers -> origin/malfet/be-remove-misisng-neon-headers 2025-12-04T08:57:44.5507906Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-12-04T08:57:44.5509685Z * [new branch] manuel/aoti_metal_shimify-thread_safe -> origin/manuel/aoti_metal_shimify-thread_safe 2025-12-04T08:57:44.5510588Z * [new branch] manuel/inductor_link_openmp -> origin/manuel/inductor_link_openmp 2025-12-04T08:57:44.5512159Z * [new branch] masnesral/metaconda -> origin/masnesral/metaconda 2025-12-04T08:57:44.5513368Z * [new branch] mem_profiler_flaky_fix -> origin/mem_profiler_flaky_fix 2025-12-04T08:57:44.5514553Z * [new branch] mem_profiler_stack_trace -> origin/mem_profiler_stack_trace 2025-12-04T08:57:44.5515708Z * [new branch] memory_profiler_stack -> origin/memory_profiler_stack 2025-12-04T08:57:44.5516941Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-12-04T08:57:44.5518081Z * [new branch] mingw_posix -> origin/mingw_posix 2025-12-04T08:57:44.5519800Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-12-04T08:57:44.5520826Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-12-04T08:57:44.5521859Z * [new branch] mlazos/acts -> origin/mlazos/acts 2025-12-04T08:57:44.5522927Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-12-04T08:57:44.5524486Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-12-04T08:57:44.5525645Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-12-04T08:57:44.5526688Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-12-04T08:57:44.5527757Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-12-04T08:57:44.5528753Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-12-04T08:57:44.5530034Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-12-04T08:57:44.5531348Z * [new branch] mlazos/bwd -> origin/mlazos/bwd 2025-12-04T08:57:44.5532825Z * [new branch] mlazos/combo-test -> origin/mlazos/combo-test 2025-12-04T08:57:44.5533992Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-12-04T08:57:44.5535109Z * [new branch] mlazos/cuda-cmd-log -> origin/mlazos/cuda-cmd-log 2025-12-04T08:57:44.5536443Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-12-04T08:57:44.5537626Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-12-04T08:57:44.5538903Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-12-04T08:57:44.5539997Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-12-04T08:57:44.5541205Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-12-04T08:57:44.5542077Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-12-04T08:57:44.5543374Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-12-04T08:57:44.5544411Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-12-04T08:57:44.5545545Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-12-04T08:57:44.5546592Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-12-04T08:57:44.5548117Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-12-04T08:57:44.5549266Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-12-04T08:57:44.5550440Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-12-04T08:57:44.5551531Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-12-04T08:57:44.5552610Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-12-04T08:57:44.5553773Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-12-04T08:57:44.5554927Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-12-04T08:57:44.5555993Z * [new branch] mlazos/fp8-fixes -> origin/mlazos/fp8-fixes 2025-12-04T08:57:44.5557104Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-12-04T08:57:44.5558180Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-12-04T08:57:44.5559377Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-12-04T08:57:44.5560698Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-12-04T08:57:44.5561920Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-12-04T08:57:44.5563090Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-12-04T08:57:44.5564154Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-12-04T08:57:44.5565218Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-12-04T08:57:44.5566275Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-12-04T08:57:44.5567333Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-12-04T08:57:44.5568845Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-12-04T08:57:44.5570039Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-12-04T08:57:44.5571138Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-12-04T08:57:44.5572253Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-12-04T08:57:44.5573324Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-12-04T08:57:44.5574363Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-12-04T08:57:44.5575441Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-12-04T08:57:44.5576557Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-12-04T08:57:44.5577651Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-12-04T08:57:44.5578726Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-12-04T08:57:44.5579883Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-12-04T08:57:44.5581006Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-12-04T08:57:44.5581969Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-12-04T08:57:44.5583212Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-12-04T08:57:44.5584134Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-12-04T08:57:44.5585224Z * [new branch] mlazos/inductor-streams -> origin/mlazos/inductor-streams 2025-12-04T08:57:44.5586107Z * [new branch] mlazos/main -> origin/mlazos/main 2025-12-04T08:57:44.5587222Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-12-04T08:57:44.5588765Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-12-04T08:57:44.5590298Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-12-04T08:57:44.5591427Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-12-04T08:57:44.5592519Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-12-04T08:57:44.5593687Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-12-04T08:57:44.5594849Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-12-04T08:57:44.5595869Z * [new branch] mlazos/overguarding -> origin/mlazos/overguarding 2025-12-04T08:57:44.5597311Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-12-04T08:57:44.5598350Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-12-04T08:57:44.5599480Z * [new branch] mlazos/resnet-fix -> origin/mlazos/resnet-fix 2025-12-04T08:57:44.5600693Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-12-04T08:57:44.5601792Z * [new branch] mlazos/rm-code -> origin/mlazos/rm-code 2025-12-04T08:57:44.5602889Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-12-04T08:57:44.5603993Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-12-04T08:57:44.5605186Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-12-04T08:57:44.5606335Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-12-04T08:57:44.5607171Z * [new branch] mlazos/stests -> origin/mlazos/stests 2025-12-04T08:57:44.5608722Z * [new branch] mlazos/stream-ops -> origin/mlazos/stream-ops 2025-12-04T08:57:44.5609803Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-12-04T08:57:44.5610960Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-12-04T08:57:44.5611951Z * [new branch] mlazos/test -> origin/mlazos/test 2025-12-04T08:57:44.5613056Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-12-04T08:57:44.5614175Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-12-04T08:57:44.5615394Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-12-04T08:57:44.5616614Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-12-04T08:57:44.5617730Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-12-04T08:57:44.5618819Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-12-04T08:57:44.5619895Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-12-04T08:57:44.5620974Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-12-04T08:57:44.5622049Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-12-04T08:57:44.5623228Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-12-04T08:57:44.5627011Z * [new branch] mlazos/user-stream-base -> origin/mlazos/user-stream-base 2025-12-04T08:57:44.5628367Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-12-04T08:57:44.5629554Z * [new branch] mlazos/user-streams-backup -> origin/mlazos/user-streams-backup 2025-12-04T08:57:44.5630719Z * [new branch] mlazos/user-streams-backup2 -> origin/mlazos/user-streams-backup2 2025-12-04T08:57:44.5631797Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-12-04T08:57:44.5632925Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-12-04T08:57:44.5634042Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-12-04T08:57:44.5635286Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-12-04T08:57:44.5636545Z * [new branch] module-shim -> origin/module-shim 2025-12-04T08:57:44.5637743Z * [new branch] move_config -> origin/move_config 2025-12-04T08:57:44.5639330Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-12-04T08:57:44.5640783Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-12-04T08:57:44.5642735Z * [new branch] mwizak/fix-triton-block-shape -> origin/mwizak/fix-triton-block-shape 2025-12-04T08:57:44.5643793Z * [new branch] my_varlen_backup -> origin/my_varlen_backup 2025-12-04T08:57:44.5644972Z * [new branch] nativert_num_outputs -> origin/nativert_num_outputs 2025-12-04T08:57:44.5646099Z * [new branch] new-codegen -> origin/new-codegen 2025-12-04T08:57:44.5647324Z * [new branch] newtest-base -> origin/newtest-base 2025-12-04T08:57:44.5648761Z * [new branch] ngimel/addmm_dtype -> origin/ngimel/addmm_dtype 2025-12-04T08:57:44.5649769Z * [new branch] ngimel/div_inv -> origin/ngimel/div_inv 2025-12-04T08:57:44.5650805Z * [new branch] ngimel/error_index_list -> origin/ngimel/error_index_list 2025-12-04T08:57:44.5651898Z * [new branch] ngimel/gather_grid -> origin/ngimel/gather_grid 2025-12-04T08:57:44.5653057Z * [new branch] ngimel/gather_grid_release -> origin/ngimel/gather_grid_release 2025-12-04T08:57:44.5653986Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-12-04T08:57:44.5655072Z * [new branch] ngimel/hostalloc -> origin/ngimel/hostalloc 2025-12-04T08:57:44.5656060Z * [new branch] ngimel/storage_id -> origin/ngimel/storage_id 2025-12-04T08:57:44.5657213Z * [new branch] nightly -> origin/nightly 2025-12-04T08:57:44.5658893Z * [new branch] nikitaved/addmm_1_rowcol_lt_path_check -> origin/nikitaved/addmm_1_rowcol_lt_path_check 2025-12-04T08:57:44.5660197Z * [new branch] nikitaved/addmm_epilogue_fusions_2d_bias -> origin/nikitaved/addmm_epilogue_fusions_2d_bias 2025-12-04T08:57:44.5661223Z * [new branch] nikitaved/addmm_epilogue_fusions_inductor -> origin/nikitaved/addmm_epilogue_fusions_inductor 2025-12-04T08:57:44.5662542Z * [new branch] nikitaved/addmm_epilogue_fusions_scratch -> origin/nikitaved/addmm_epilogue_fusions_scratch 2025-12-04T08:57:44.5663902Z * [new branch] nikitaved/grad_addmm_epilogue_fusions -> origin/nikitaved/grad_addmm_epilogue_fusions 2025-12-04T08:57:44.5665337Z * [new branch] nikitaved/simpler_can_use_32bit_index -> origin/nikitaved/simpler_can_use_32bit_index 2025-12-04T08:57:44.5666279Z * [new branch] nikitaved/test -> origin/nikitaved/test 2025-12-04T08:57:44.5668363Z * [new branch] nmacchioni-perf-test-async-autotune -> origin/nmacchioni-perf-test-async-autotune 2025-12-04T08:57:44.5669361Z * [new branch] no_distributed_log_spew -> origin/no_distributed_log_spew 2025-12-04T08:57:44.5670686Z * [new branch] nofun-hack -> origin/nofun-hack 2025-12-04T08:57:44.5671838Z * [new branch] norm_bench -> origin/norm_bench 2025-12-04T08:57:44.5673484Z * [new branch] nullplay/fuse_matmul -> origin/nullplay/fuse_matmul 2025-12-04T08:57:44.5674700Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-12-04T08:57:44.5675890Z * [new branch] optimizer_test -> origin/optimizer_test 2025-12-04T08:57:44.5677748Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-12-04T08:57:44.5678918Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-12-04T08:57:44.5680200Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-12-04T08:57:44.5681552Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-12-04T08:57:44.5682733Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-12-04T08:57:44.5683991Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-12-04T08:57:44.5685145Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-12-04T08:57:44.5686276Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-12-04T08:57:44.5687477Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-12-04T08:57:44.5688589Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-12-04T08:57:44.5689687Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-12-04T08:57:44.5690782Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-12-04T08:57:44.5691821Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-12-04T08:57:44.5692878Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-12-04T08:57:44.5693965Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-12-04T08:57:44.5695305Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-12-04T08:57:44.5697224Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-12-04T08:57:44.5698248Z * [new branch] orig/release/2.9 -> origin/orig/release/2.9 2025-12-04T08:57:44.5700616Z * [new branch] origin/gh/fxdawnn/1/base -> origin/origin/gh/fxdawnn/1/base 2025-12-04T08:57:44.5701615Z * [new branch] origin/gh/fxdawnn/1/orig -> origin/origin/gh/fxdawnn/1/orig 2025-12-04T08:57:44.5703978Z * [new branch] origin/gh/zpcore/14/orig -> origin/origin/gh/zpcore/14/orig 2025-12-04T08:57:44.5705282Z * [new branch] oulgen-patch-1 -> origin/oulgen-patch-1 2025-12-04T08:57:44.5706576Z * [new branch] oulgen-patch-2 -> origin/oulgen-patch-2 2025-12-04T08:57:44.5708166Z * [new branch] oulgen-patch-3 -> origin/oulgen-patch-3 2025-12-04T08:57:44.5709851Z * [new branch] oulgen-patch-4 -> origin/oulgen-patch-4 2025-12-04T08:57:44.5711033Z * [new branch] padded-tensor -> origin/padded-tensor 2025-12-04T08:57:44.5712262Z * [new branch] pca2 -> origin/pca2 2025-12-04T08:57:44.5713588Z * [new branch] per_channel_backup -> origin/per_channel_backup 2025-12-04T08:57:44.5714767Z * [new branch] perf_ops -> origin/perf_ops 2025-12-04T08:57:44.5715961Z * [new branch] perf_ops_2_9 -> origin/perf_ops_2_9 2025-12-04T08:57:44.5717843Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-12-04T08:57:44.5719274Z * [new branch] pianpwk/__draft_debug_mode -> origin/pianpwk/__draft_debug_mode 2025-12-04T08:57:44.5720529Z * [new branch] pianpwk/_debug_mode_for_triton_draft -> origin/pianpwk/_debug_mode_for_triton_draft 2025-12-04T08:57:44.5722028Z * [new branch] pianpwk/_debug_nn_module_compile -> origin/pianpwk/_debug_nn_module_compile 2025-12-04T08:57:44.5723010Z * [new branch] pianpwk/_draft_triton_11_3 -> origin/pianpwk/_draft_triton_11_3 2025-12-04T08:57:44.5724164Z * [new branch] pianpwk/_manual_bucket_draft -> origin/pianpwk/_manual_bucket_draft 2025-12-04T08:57:44.5725797Z * [new branch] pianpwk/_profile_w_dispatch_keys -> origin/pianpwk/_profile_w_dispatch_keys 2025-12-04T08:57:44.5727242Z * [new branch] pianpwk/_super_draft_debug_mode -> origin/pianpwk/_super_draft_debug_mode 2025-12-04T08:57:44.5728668Z * [new branch] pianpwk/_unbacked_local_shard_size -> origin/pianpwk/_unbacked_local_shard_size 2025-12-04T08:57:44.5729819Z * [new branch] pianpwk/anomaly_tb -> origin/pianpwk/anomaly_tb 2025-12-04T08:57:44.5730936Z * [new branch] pianpwk/auto_fx_annotate -> origin/pianpwk/auto_fx_annotate 2025-12-04T08:57:44.5732137Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-12-04T08:57:44.5733150Z * [new branch] pianpwk/bert_dynamic_perf -> origin/pianpwk/bert_dynamic_perf 2025-12-04T08:57:44.5734360Z * [new branch] pianpwk/debug_fwd_stack_traces -> origin/pianpwk/debug_fwd_stack_traces 2025-12-04T08:57:44.5735540Z * [new branch] pianpwk/debug_hash_tensor -> origin/pianpwk/debug_hash_tensor 2025-12-04T08:57:44.5736808Z * [new branch] pianpwk/debug_mode_annotate -> origin/pianpwk/debug_mode_annotate 2025-12-04T08:57:44.5737841Z * [new branch] pianpwk/debug_mode_defaults -> origin/pianpwk/debug_mode_defaults 2025-12-04T08:57:44.5738910Z * [new branch] pianpwk/debug_mode_hacks -> origin/pianpwk/debug_mode_hacks 2025-12-04T08:57:44.5740050Z * [new branch] pianpwk/debug_mode_opcall_refactor -> origin/pianpwk/debug_mode_opcall_refactor 2025-12-04T08:57:44.5741188Z * [new branch] pianpwk/debug_mode_show_ids -> origin/pianpwk/debug_mode_show_ids 2025-12-04T08:57:44.5742266Z * [new branch] pianpwk/debug_mode_triton -> origin/pianpwk/debug_mode_triton 2025-12-04T08:57:44.5743527Z * [new branch] pianpwk/debug_show_stack_trace -> origin/pianpwk/debug_show_stack_trace 2025-12-04T08:57:44.5744701Z * [new branch] pianpwk/debug_wait_on_collective -> origin/pianpwk/debug_wait_on_collective 2025-12-04T08:57:44.5745775Z * [new branch] pianpwk/debugmode_compile_tf -> origin/pianpwk/debugmode_compile_tf 2025-12-04T08:57:44.5747055Z * [new branch] pianpwk/dispatch_key_debugging_for_debug -> origin/pianpwk/dispatch_key_debugging_for_debug 2025-12-04T08:57:44.5748392Z * [new branch] pianpwk/draft_debug_mode_tfcompile -> origin/pianpwk/draft_debug_mode_tfcompile 2025-12-04T08:57:44.5749511Z * [new branch] pianpwk/draft_multikernel_nn -> origin/pianpwk/draft_multikernel_nn 2025-12-04T08:57:44.5750694Z * [new branch] pianpwk/draft_multikernel_status_10_5 -> origin/pianpwk/draft_multikernel_status_10_5 2025-12-04T08:57:44.5751840Z * [new branch] pianpwk/dtensor_custom_chunk -> origin/pianpwk/dtensor_custom_chunk 2025-12-04T08:57:44.5753112Z * [new branch] pianpwk/dtensor_unbacked_keypath -> origin/pianpwk/dtensor_unbacked_keypath 2025-12-04T08:57:44.5754233Z * [new branch] pianpwk/event_list_tree -> origin/pianpwk/event_list_tree 2025-12-04T08:57:44.5755575Z * [new branch] pianpwk/false_numel_refs -> origin/pianpwk/false_numel_refs 2025-12-04T08:57:44.5756572Z * [new branch] pianpwk/maybe_guard_rel -> origin/pianpwk/maybe_guard_rel 2025-12-04T08:57:44.5757732Z * [new branch] pianpwk/multikernel_hints_draft -> origin/pianpwk/multikernel_hints_draft 2025-12-04T08:57:44.5758980Z * [new branch] pianpwk/no_size_oblivious_slice_scat -> origin/pianpwk/no_size_oblivious_slice_scat 2025-12-04T08:57:44.5760342Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-12-04T08:57:44.5761318Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-12-04T08:57:44.5762472Z * [new branch] pianpwk/skip_python_keys_alternate -> origin/pianpwk/skip_python_keys_alternate 2025-12-04T08:57:44.5763550Z * [new branch] pianpwk/skip_python_keys_in_guards -> origin/pianpwk/skip_python_keys_in_guards 2025-12-04T08:57:44.5764580Z * [new branch] pianpwk/sym_tokens_draft -> origin/pianpwk/sym_tokens_draft 2025-12-04T08:57:44.5765635Z * [new branch] pianpwk/symint_one_hot -> origin/pianpwk/symint_one_hot 2025-12-04T08:57:44.5766908Z * [new branch] pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false 2025-12-04T08:57:44.5767913Z * [new branch] pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap 2025-12-04T08:57:44.5768952Z * [new branch] pianpwk/try_dumb_stuff -> origin/pianpwk/try_dumb_stuff 2025-12-04T08:57:44.5770019Z * [new branch] pianpwk/try_dumb_stuff_2 -> origin/pianpwk/try_dumb_stuff_2 2025-12-04T08:57:44.5771205Z * [new branch] pianpwk/unbacked_dtensor_mm -> origin/pianpwk/unbacked_dtensor_mm 2025-12-04T08:57:44.5772291Z * [new branch] pianpwk/unbacked_tracing_12_2 -> origin/pianpwk/unbacked_tracing_12_2 2025-12-04T08:57:44.5773328Z * [new branch] pianpwk/user_symints -> origin/pianpwk/user_symints 2025-12-04T08:57:44.5774486Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-12-04T08:57:44.5775990Z * [new branch] piz/fix_partial_backward_1112 -> origin/piz/fix_partial_backward_1112 2025-12-04T08:57:44.5777070Z * [new branch] piz/prop_cache_clean -> origin/piz/prop_cache_clean 2025-12-04T08:57:44.5778296Z * [new branch] pool-separate -> origin/pool-separate 2025-12-04T08:57:44.5779425Z * [new branch] pr-156087 -> origin/pr-156087 2025-12-04T08:57:44.5780926Z * [new branch] pr/131860 -> origin/pr/131860 2025-12-04T08:57:44.5782106Z * [new branch] predispatch_to -> origin/predispatch_to 2025-12-04T08:57:44.5783236Z * [new branch] protect-c17 -> origin/protect-c17 2025-12-04T08:57:44.5784395Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-12-04T08:57:44.5786075Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-12-04T08:57:44.5787977Z * [new branch] q1l1/fix_device_moved_constant_type_unknown -> origin/q1l1/fix_device_moved_constant_type_unknown 2025-12-04T08:57:44.5789452Z * [new branch] q1l1/fix_wrong_default_type_for_kernel_call_args -> origin/q1l1/fix_wrong_default_type_for_kernel_call_args 2025-12-04T08:57:44.5791738Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-12-04T08:57:44.5793098Z * [new branch] quote-pytest_cache -> origin/quote-pytest_cache 2025-12-04T08:57:44.5794609Z * [new branch] reland-accgrad-stream-warn -> origin/reland-accgrad-stream-warn 2025-12-04T08:57:44.5796336Z * [new branch] release/1.10 -> origin/release/1.10 2025-12-04T08:57:44.5797592Z * [new branch] release/1.11 -> origin/release/1.11 2025-12-04T08:57:44.5798763Z * [new branch] release/1.12 -> origin/release/1.12 2025-12-04T08:57:44.5800002Z * [new branch] release/1.13 -> origin/release/1.13 2025-12-04T08:57:44.5801076Z * [new branch] release/1.4 -> origin/release/1.4 2025-12-04T08:57:44.5801994Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-12-04T08:57:44.5803101Z * [new branch] release/1.5 -> origin/release/1.5 2025-12-04T08:57:44.5804296Z * [new branch] release/1.6 -> origin/release/1.6 2025-12-04T08:57:44.5805412Z * [new branch] release/1.7 -> origin/release/1.7 2025-12-04T08:57:44.5806642Z * [new branch] release/1.8 -> origin/release/1.8 2025-12-04T08:57:44.5807727Z * [new branch] release/1.9 -> origin/release/1.9 2025-12-04T08:57:44.5808879Z * [new branch] release/2.0 -> origin/release/2.0 2025-12-04T08:57:44.5810063Z * [new branch] release/2.1 -> origin/release/2.1 2025-12-04T08:57:44.5811274Z * [new branch] release/2.2 -> origin/release/2.2 2025-12-04T08:57:44.5812725Z * [new branch] release/2.3 -> origin/release/2.3 2025-12-04T08:57:44.5814288Z * [new branch] release/2.4 -> origin/release/2.4 2025-12-04T08:57:44.5815848Z * [new branch] release/2.5 -> origin/release/2.5 2025-12-04T08:57:44.5817059Z * [new branch] release/2.6 -> origin/release/2.6 2025-12-04T08:57:44.5818266Z * [new branch] release/2.7 -> origin/release/2.7 2025-12-04T08:57:44.5819616Z * [new branch] release/2.8 -> origin/release/2.8 2025-12-04T08:57:44.5820820Z * [new branch] release/2.9 -> origin/release/2.9 2025-12-04T08:57:44.5822050Z * [new branch] release_notes -> origin/release_notes 2025-12-04T08:57:44.5823310Z * [new branch] remove_pyinterpreter -> origin/remove_pyinterpreter 2025-12-04T08:57:44.5827801Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-12-04T08:57:44.5829449Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-12-04T08:57:44.5830578Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-12-04T08:57:44.5831787Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-12-04T08:57:44.5834110Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-12-04T08:57:44.5836334Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-12-04T08:57:44.5838477Z * [new branch] revert-152361-gh/fadara01/1/head -> origin/revert-152361-gh/fadara01/1/head 2025-12-04T08:57:44.5840978Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-12-04T08:57:44.5842547Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-12-04T08:57:44.5843467Z * [new branch] revert-hoo-invoke-subgraph -> origin/revert-hoo-invoke-subgraph 2025-12-04T08:57:44.5844746Z * [new branch] revert_always_build_distributed -> origin/revert_always_build_distributed 2025-12-04T08:57:44.5845816Z * [new branch] rms_norm_patch -> origin/rms_norm_patch 2025-12-04T08:57:44.5847423Z * [new branch] ruisi/fix_all_to_all_estimation -> origin/ruisi/fix_all_to_all_estimation 2025-12-04T08:57:44.5848552Z * [new branch] ruisi/fix_comm_estimation -> origin/ruisi/fix_comm_estimation 2025-12-04T08:57:44.5849554Z * [new branch] ruisi/fix_dynamic_shape_estimation -> origin/ruisi/fix_dynamic_shape_estimation 2025-12-04T08:57:44.5850615Z * [new branch] ruisi/fix_llama3_autobucketing -> origin/ruisi/fix_llama3_autobucketing 2025-12-04T08:57:44.5852032Z * [new branch] ruisi/fix_manual_bucketing_ep_pass -> origin/ruisi/fix_manual_bucketing_ep_pass 2025-12-04T08:57:44.5853391Z * [new branch] ruisi/manual_bucket_pass -> origin/ruisi/manual_bucket_pass 2025-12-04T08:57:44.5855283Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-12-04T08:57:44.5856136Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-12-04T08:57:44.5857646Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-12-04T08:57:44.5858742Z * [new branch] rzou/njt -> origin/rzou/njt 2025-12-04T08:57:44.5859906Z * [new branch] rzou/pca -> origin/rzou/pca 2025-12-04T08:57:44.5860972Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-12-04T08:57:44.5862147Z * [new branch] samplevllm -> origin/samplevllm 2025-12-04T08:57:44.5864073Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-12-04T08:57:44.5865104Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-12-04T08:57:44.5866693Z * [new branch] sapling-pr-archive-tushar00jain -> origin/sapling-pr-archive-tushar00jain 2025-12-04T08:57:44.5868020Z * [new branch] save -> origin/save 2025-12-04T08:57:44.5869327Z * [new branch] scaled_mm -> origin/scaled_mm 2025-12-04T08:57:44.5870538Z * [new branch] scan_attempt -> origin/scan_attempt 2025-12-04T08:57:44.5872145Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-12-04T08:57:44.5873451Z * [new branch] sekyondaMeta-dynamoconfig-fix -> origin/sekyondaMeta-dynamoconfig-fix 2025-12-04T08:57:44.5874906Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-12-04T08:57:44.5876203Z * [new branch] shoumikhin-patch-1 -> origin/shoumikhin-patch-1 2025-12-04T08:57:44.5877432Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-12-04T08:57:44.5878603Z * [new branch] some_rocm_inductor_skips -> origin/some_rocm_inductor_skips 2025-12-04T08:57:44.5880285Z * [new branch] soulitzer/stash-tls-ac -> origin/soulitzer/stash-tls-ac 2025-12-04T08:57:44.5881548Z * [new branch] sparse-mm-bf16-support -> origin/sparse-mm-bf16-support 2025-12-04T08:57:44.5882745Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-12-04T08:57:44.5883895Z * [new branch] suo -> origin/suo 2025-12-04T08:57:44.5885067Z * [new branch] sve-poc -> origin/sve-poc 2025-12-04T08:57:44.5886299Z * [new branch] switch-bn -> origin/switch-bn 2025-12-04T08:57:44.5887969Z * [new branch] sy_annotation_in_autograd_hop -> origin/sy_annotation_in_autograd_hop 2025-12-04T08:57:44.5889109Z * [new branch] sy_aot_eager_record -> origin/sy_aot_eager_record 2025-12-04T08:57:44.5890375Z * [new branch] sy_custom_bucketing -> origin/sy_custom_bucketing 2025-12-04T08:57:44.5891522Z * [new branch] sy_debug_mode_test -> origin/sy_debug_mode_test 2025-12-04T08:57:44.5892838Z * [new branch] sy_deserialize -> origin/sy_deserialize 2025-12-04T08:57:44.5893889Z * [new branch] sy_dump_gm_code -> origin/sy_dump_gm_code 2025-12-04T08:57:44.5895088Z * [new branch] sy_exp -> origin/sy_exp 2025-12-04T08:57:44.5896300Z * [new branch] sy_export_annotation -> origin/sy_export_annotation 2025-12-04T08:57:44.5897437Z * [new branch] sy_invoke_subgraph -> origin/sy_invoke_subgraph 2025-12-04T08:57:44.5898578Z * [new branch] sy_kernel_bw_name -> origin/sy_kernel_bw_name 2025-12-04T08:57:44.5899693Z * [new branch] sy_multi_arch -> origin/sy_multi_arch 2025-12-04T08:57:44.5900882Z * [new branch] sy_nn_module_stack -> origin/sy_nn_module_stack 2025-12-04T08:57:44.5902027Z * [new branch] sy_original_dtensor -> origin/sy_original_dtensor 2025-12-04T08:57:44.5903185Z * [new branch] sy_profiler_cia -> origin/sy_profiler_cia 2025-12-04T08:57:44.5904331Z * [new branch] symm_mem_sync -> origin/symm_mem_sync 2025-12-04T08:57:44.5905559Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-12-04T08:57:44.5906792Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-12-04T08:57:44.5908360Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-12-04T08:57:44.5909630Z * [new branch] test-old -> origin/test-old 2025-12-04T08:57:44.5911223Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-12-04T08:57:44.5912860Z * [new branch] tianren/customOp_autotune_fix -> origin/tianren/customOp_autotune_fix 2025-12-04T08:57:44.5914018Z * [new branch] tianren/customOp_enable_max_autotune -> origin/tianren/customOp_enable_max_autotune 2025-12-04T08:57:44.5915015Z * [new branch] tianren/customOp_fusion -> origin/tianren/customOp_fusion 2025-12-04T08:57:44.5916258Z * [new branch] tianren/customop_collectiveop_benchmark -> origin/tianren/customop_collectiveop_benchmark 2025-12-04T08:57:44.5917627Z * [new branch] tianren/customop_collectiveop_benchmark_fix -> origin/tianren/customop_collectiveop_benchmark_fix 2025-12-04T08:57:44.5918967Z * [new branch] tianren/customop_dynamic_config -> origin/tianren/customop_dynamic_config 2025-12-04T08:57:44.5920195Z * [new branch] tianren/dynamic_range_input -> origin/tianren/dynamic_range_input 2025-12-04T08:57:44.5921372Z * [new branch] tianren/dynamic_range_input_fix -> origin/tianren/dynamic_range_input_fix 2025-12-04T08:57:44.5922431Z * [new branch] tianren/dynamic_range_input_merge -> origin/tianren/dynamic_range_input_merge 2025-12-04T08:57:44.5924072Z * [new branch] tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp 2025-12-04T08:57:44.5925528Z * [new branch] tianren/fx_codegen_dump -> origin/tianren/fx_codegen_dump 2025-12-04T08:57:44.5926646Z * [new branch] tianren/symmetric_memory -> origin/tianren/symmetric_memory 2025-12-04T08:57:44.5927796Z * [new branch] tianren/test -> origin/tianren/test 2025-12-04T08:57:44.5929065Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-12-04T08:57:44.5930220Z * [new branch] tmp -> origin/tmp 2025-12-04T08:57:44.5931665Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-12-04T08:57:44.5932981Z * [new branch] torchtitan_integration -> origin/torchtitan_integration 2025-12-04T08:57:44.5934239Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-12-04T08:57:44.5935482Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-12-04T08:57:44.5936712Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-12-04T08:57:44.5938606Z * [new branch] triton_kernel -> origin/triton_kernel 2025-12-04T08:57:44.5939953Z * [new branch] tt_pkg_1908 -> origin/tt_pkg_1908 2025-12-04T08:57:44.5940731Z * [new branch] type_dec -> origin/type_dec 2025-12-04T08:57:44.5941609Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-12-04T08:57:44.5943240Z * [new branch] update-audio-commit-hash/17630256502-1803-1 -> origin/update-audio-commit-hash/17630256502-1803-1 2025-12-04T08:57:44.5944241Z * [new branch] update-audio-commit-hash/19087141161-1916-1 -> origin/update-audio-commit-hash/19087141161-1916-1 2025-12-04T08:57:44.5945383Z * [new branch] update-audio-commit-hash/19250643381-1929-1 -> origin/update-audio-commit-hash/19250643381-1929-1 2025-12-04T08:57:44.5946569Z * [new branch] update-audio-commit-hash/19397724337-1935-1 -> origin/update-audio-commit-hash/19397724337-1935-1 2025-12-04T08:57:44.5947575Z * [new branch] update-audio-commit-hash/19555670148-1941-1 -> origin/update-audio-commit-hash/19555670148-1941-1 2025-12-04T08:57:44.5949344Z * [new branch] update-audio-commit-hash/19750627930-1946-1 -> origin/update-audio-commit-hash/19750627930-1946-1 2025-12-04T08:57:44.5950941Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-12-04T08:57:44.5952526Z * [new branch] update-vision-commit-hash/19087141161-1916-1 -> origin/update-vision-commit-hash/19087141161-1916-1 2025-12-04T08:57:44.5953595Z * [new branch] update-vision-commit-hash/19184897099-1925-1 -> origin/update-vision-commit-hash/19184897099-1925-1 2025-12-04T08:57:44.5954619Z * [new branch] update-vision-commit-hash/19250643381-1929-1 -> origin/update-vision-commit-hash/19250643381-1929-1 2025-12-04T08:57:44.5955804Z * [new branch] update-vision-commit-hash/19381328640-1934-1 -> origin/update-vision-commit-hash/19381328640-1934-1 2025-12-04T08:57:44.5956809Z * [new branch] update-vision-commit-hash/19485237164-1938-1 -> origin/update-vision-commit-hash/19485237164-1938-1 2025-12-04T08:57:44.5958898Z * [new branch] update-vllm-commit-hash/18451675449-1879-1 -> origin/update-vllm-commit-hash/18451675449-1879-1 2025-12-04T08:57:44.5960112Z * [new branch] update-vllm-dockerfile -> origin/update-vllm-dockerfile 2025-12-04T08:57:44.5961783Z * [new branch] update-xla-commit-hash/19224287370-211-1 -> origin/update-xla-commit-hash/19224287370-211-1 2025-12-04T08:57:44.5962933Z * [new branch] update-xla-commit-hash/19422028566-212-1 -> origin/update-xla-commit-hash/19422028566-212-1 2025-12-04T08:57:44.5964002Z * [new branch] update-xla-commit-hash/19626841311-213-1 -> origin/update-xla-commit-hash/19626841311-213-1 2025-12-04T08:57:44.5965255Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-12-04T08:57:44.5966302Z * [new branch] update_operator_readme -> origin/update_operator_readme 2025-12-04T08:57:44.5967522Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-12-04T08:57:44.5968658Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-12-04T08:57:44.5969934Z * [new branch] update_slow_tests_1762155677 -> origin/update_slow_tests_1762155677 2025-12-04T08:57:44.5971108Z * [new branch] update_slow_tests_1763365283 -> origin/update_slow_tests_1763365283 2025-12-04T08:57:44.5972441Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-12-04T08:57:44.5973471Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-12-04T08:57:44.5974649Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-12-04T08:57:44.5975876Z * [new branch] upload-tests-for-autorevert -> origin/upload-tests-for-autorevert 2025-12-04T08:57:44.5977025Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-12-04T08:57:44.5978314Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-12-04T08:57:44.5979560Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-12-04T08:57:44.5980825Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-12-04T08:57:44.5982376Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-12-04T08:57:44.5983580Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-12-04T08:57:44.5984822Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-12-04T08:57:44.5986026Z * [new branch] validate_fn -> origin/validate_fn 2025-12-04T08:57:44.5987398Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-12-04T08:57:44.5989081Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-12-04T08:57:44.5990295Z * [new branch] varlen-api -> origin/varlen-api 2025-12-04T08:57:44.5991541Z * [new branch] varlen-api-backup -> origin/varlen-api-backup 2025-12-04T08:57:44.5992681Z * [new branch] varlen_batch_invariance -> origin/varlen_batch_invariance 2025-12-04T08:57:44.5994052Z * [new branch] viable/strict -> origin/viable/strict 2025-12-04T08:57:44.5995993Z * [new branch] vishal9-team/dtensor_parallelism_toy -> origin/vishal9-team/dtensor_parallelism_toy 2025-12-04T08:57:44.5997093Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-12-04T08:57:44.5998375Z * [new branch] vllmpin -> origin/vllmpin 2025-12-04T08:57:44.5999887Z * [new branch] vscode-recommend-pyrefly -> origin/vscode-recommend-pyrefly 2025-12-04T08:57:44.6001214Z * [new branch] wdvr-patch-1 -> origin/wdvr-patch-1 2025-12-04T08:57:44.6002671Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-12-04T08:57:44.6004111Z * [new branch] whc/pei -> origin/whc/pei 2025-12-04T08:57:44.6005184Z * [new branch] whc/pp_fix -> origin/whc/pp_fix 2025-12-04T08:57:44.6006449Z * [new branch] whc/sharding -> origin/whc/sharding 2025-12-04T08:57:44.6007492Z * [new branch] whc/sharding2 -> origin/whc/sharding2 2025-12-04T08:57:44.6008510Z * [new branch] whc/uneven -> origin/whc/uneven 2025-12-04T08:57:44.6010440Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-12-04T08:57:44.6011618Z * [new branch] win_warnings -> origin/win_warnings 2025-12-04T08:57:44.6012802Z * [new branch] windows_libtorch_free -> origin/windows_libtorch_free 2025-12-04T08:57:44.6013931Z * [new branch] xmfan-war -> origin/xmfan-war 2025-12-04T08:57:44.6015449Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-12-04T08:57:44.6016585Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-12-04T08:57:44.6017918Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-12-04T08:57:44.6018640Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-12-04T08:57:44.6019763Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-12-04T08:57:44.6020633Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-12-04T08:57:44.6021732Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-12-04T08:57:44.6023017Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-12-04T08:57:44.6025407Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-12-04T08:57:44.6026522Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-12-04T08:57:44.6027797Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-12-04T08:57:44.6028793Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-12-04T08:57:44.6029908Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-12-04T08:57:44.6031028Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-12-04T08:57:44.6032198Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-12-04T08:57:44.6033323Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-12-04T08:57:44.6034498Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-12-04T08:57:44.6035556Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-12-04T08:57:44.6036749Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-12-04T08:57:44.6037846Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-12-04T08:57:44.6039037Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-12-04T08:57:44.6040381Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-12-04T08:57:44.6041711Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T08:57:44.6042720Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-12-04T08:57:44.6043558Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-12-04T08:57:44.6044697Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-12-04T08:57:44.6045819Z * [new branch] xmfan/test -> origin/xmfan/test 2025-12-04T08:57:44.6047851Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-12-04T08:57:44.6048905Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-12-04T08:57:44.6050027Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-12-04T08:57:44.6051373Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-12-04T08:57:44.6052534Z * [new branch] yiming/run_with_start_end_rng_hop -> origin/yiming/run_with_start_end_rng_hop 2025-12-04T08:57:44.6053724Z * [new branch] yolo-llama3 -> origin/yolo-llama3 2025-12-04T08:57:44.6055202Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-12-04T08:57:44.6056877Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-12-04T08:57:44.6057914Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-12-04T08:57:44.6058984Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-12-04T08:57:44.6060282Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-12-04T08:57:44.6061558Z * [new branch] zb2p -> origin/zb2p 2025-12-04T08:57:44.6063118Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-12-04T08:57:44.6064862Z * [new branch] zhxchen17/ci/vllm_lora_oom -> origin/zhxchen17/ci/vllm_lora_oom 2025-12-04T08:57:44.6065982Z * [new branch] zhxchen17/ci/vllm_multimodal_oom -> origin/zhxchen17/ci/vllm_multimodal_oom 2025-12-04T08:57:44.6066951Z * [new branch] zhxchen17/ci/vllm_pin -> origin/zhxchen17/ci/vllm_pin 2025-12-04T08:57:44.6069031Z * [new branch] zhxchen17/dynamo/unsafe_drop_all_guards -> origin/zhxchen17/dynamo/unsafe_drop_all_guards 2025-12-04T08:57:44.6070996Z * [new branch] zhxchen17/export/call_override -> origin/zhxchen17/export/call_override 2025-12-04T08:57:44.6072128Z * [new branch] zhxchen17/export/codemod1 -> origin/zhxchen17/export/codemod1 2025-12-04T08:57:44.6073319Z * [new branch] zhxchen17/export/ctx_return -> origin/zhxchen17/export/ctx_return 2025-12-04T08:57:44.6074621Z * [new branch] zhxchen17/export/disable_side_effect_warn -> origin/zhxchen17/export/disable_side_effect_warn 2025-12-04T08:57:44.6075574Z * [new branch] zhxchen17/export/pytree_check -> origin/zhxchen17/export/pytree_check 2025-12-04T08:57:44.6077062Z * [new branch] zhxchen17/precompile/aoti -> origin/zhxchen17/precompile/aoti 2025-12-04T08:57:44.6078385Z * [new branch] zhxchen17/precompile/globals -> origin/zhxchen17/precompile/globals 2025-12-04T08:57:44.6079533Z * [new branch] zhxchen17/precompile/inductor_guards -> origin/zhxchen17/precompile/inductor_guards 2025-12-04T08:57:44.6080908Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-12-04T08:57:44.6082114Z * [new branch] zhxchen17/torch_export_api_update -> origin/zhxchen17/torch_export_api_update 2025-12-04T08:57:44.6083551Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-12-04T08:57:44.6085217Z * [new branch] zxiiro/build-times -> origin/zxiiro/build-times 2025-12-04T08:57:44.6086370Z * [new branch] zxiiro/c7i.2xlarge -> origin/zxiiro/c7i.2xlarge 2025-12-04T08:57:44.6087566Z * [new branch] zxiiro/c7i.2xlarge.h100 -> origin/zxiiro/c7i.2xlarge.h100 2025-12-04T08:57:44.6088601Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-12-04T08:57:44.6089729Z * [new branch] zxiiro/risc64 -> origin/zxiiro/risc64 2025-12-04T08:57:44.6090916Z * [new branch] zxiiro/test-multicloud-arc -> origin/zxiiro/test-multicloud-arc 2025-12-04T08:57:44.6092044Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-12-04T08:57:44.6092766Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-12-04T08:57:44.6093760Z * [new tag] ciflow/b200/115316 -> ciflow/b200/115316 2025-12-04T08:57:44.6094452Z * [new tag] ciflow/b200/160685 -> ciflow/b200/160685 2025-12-04T08:57:44.6095109Z * [new tag] ciflow/b200/161607 -> ciflow/b200/161607 2025-12-04T08:57:44.6095818Z * [new tag] ciflow/b200/161938 -> ciflow/b200/161938 2025-12-04T08:57:44.6096701Z * [new tag] ciflow/b200/167207 -> ciflow/b200/167207 2025-12-04T08:57:44.6097356Z * [new tag] ciflow/b200/167989 -> ciflow/b200/167989 2025-12-04T08:57:44.6098241Z * [new tag] ciflow/b200/168096 -> ciflow/b200/168096 2025-12-04T08:57:44.6099031Z * [new tag] ciflow/b200/168175 -> ciflow/b200/168175 2025-12-04T08:57:44.6099740Z * [new tag] ciflow/b200/168195 -> ciflow/b200/168195 2025-12-04T08:57:44.6100697Z * [new tag] ciflow/b200/169200 -> ciflow/b200/169200 2025-12-04T08:57:44.6101358Z * [new tag] ciflow/b200/169216 -> ciflow/b200/169216 2025-12-04T08:57:44.6102498Z * [new tag] ciflow/b200/169380 -> ciflow/b200/169380 2025-12-04T08:57:44.6103629Z * [new tag] ciflow/b200/169412 -> ciflow/b200/169412 2025-12-04T08:57:44.6104559Z * [new tag] ciflow/b200/169470 -> ciflow/b200/169470 2025-12-04T08:57:44.6105336Z * [new tag] ciflow/b200/169471 -> ciflow/b200/169471 2025-12-04T08:57:44.6106102Z * [new tag] ciflow/b200/169472 -> ciflow/b200/169472 2025-12-04T08:57:44.6107066Z * [new tag] ciflow/b200/169514 -> ciflow/b200/169514 2025-12-04T08:57:44.6108124Z * [new tag] ciflow/b200/169517 -> ciflow/b200/169517 2025-12-04T08:57:44.6109253Z * [new tag] ciflow/binaries/165922 -> ciflow/binaries/165922 2025-12-04T08:57:44.6109945Z * [new tag] ciflow/binaries/169510 -> ciflow/binaries/169510 2025-12-04T08:57:44.6111051Z * [new tag] ciflow/binaries_wheel/157994 -> ciflow/binaries_wheel/157994 2025-12-04T08:57:44.6111863Z * [new tag] ciflow/binaries_wheel/166829 -> ciflow/binaries_wheel/166829 2025-12-04T08:57:44.6112568Z * [new tag] ciflow/binaries_wheel/167972 -> ciflow/binaries_wheel/167972 2025-12-04T08:57:44.6113364Z * [new tag] ciflow/binaries_wheel/167981 -> ciflow/binaries_wheel/167981 2025-12-04T08:57:44.6114166Z * [new tag] ciflow/dynamo/167695 -> ciflow/dynamo/167695 2025-12-04T08:57:44.6114950Z * [new tag] ciflow/dynamo/168096 -> ciflow/dynamo/168096 2025-12-04T08:57:44.6115778Z * [new tag] ciflow/dynamo/169525 -> ciflow/dynamo/169525 2025-12-04T08:57:44.6116835Z * [new tag] ciflow/h100-cutlass-backend/161938 -> ciflow/h100-cutlass-backend/161938 2025-12-04T08:57:44.6117473Z * [new tag] ciflow/h100-cutlass-backend/161940 -> ciflow/h100-cutlass-backend/161940 2025-12-04T08:57:44.6118468Z * [new tag] ciflow/h100-distributed/168923 -> ciflow/h100-distributed/168923 2025-12-04T08:57:44.6119207Z * [new tag] ciflow/h100-symm-mem/167552 -> ciflow/h100-symm-mem/167552 2025-12-04T08:57:44.6120017Z * [new tag] ciflow/h100-symm-mem/168129 -> ciflow/h100-symm-mem/168129 2025-12-04T08:57:44.6120720Z * [new tag] ciflow/h100-symm-mem/168917 -> ciflow/h100-symm-mem/168917 2025-12-04T08:57:44.6121764Z * [new tag] ciflow/h100-symm-mem/169156 -> ciflow/h100-symm-mem/169156 2025-12-04T08:57:44.6122474Z * [new tag] ciflow/h100-symm-mem/169200 -> ciflow/h100-symm-mem/169200 2025-12-04T08:57:44.6123160Z * [new tag] ciflow/h100-symm-mem/169216 -> ciflow/h100-symm-mem/169216 2025-12-04T08:57:44.6124180Z * [new tag] ciflow/h100-symm-mem/169338 -> ciflow/h100-symm-mem/169338 2025-12-04T08:57:44.6125177Z * [new tag] ciflow/h100-symm-mem/169355 -> ciflow/h100-symm-mem/169355 2025-12-04T08:57:44.6125859Z * [new tag] ciflow/h100-symm-mem/169543 -> ciflow/h100-symm-mem/169543 2025-12-04T08:57:44.6127273Z * [new tag] ciflow/h100/115316 -> ciflow/h100/115316 2025-12-04T08:57:44.6127928Z * [new tag] ciflow/h100/160685 -> ciflow/h100/160685 2025-12-04T08:57:44.6128656Z * [new tag] ciflow/h100/160729 -> ciflow/h100/160729 2025-12-04T08:57:44.6129334Z * [new tag] ciflow/h100/161607 -> ciflow/h100/161607 2025-12-04T08:57:44.6130069Z * [new tag] ciflow/h100/161938 -> ciflow/h100/161938 2025-12-04T08:57:44.6130763Z * [new tag] ciflow/h100/167207 -> ciflow/h100/167207 2025-12-04T08:57:44.6131611Z * [new tag] ciflow/h100/167989 -> ciflow/h100/167989 2025-12-04T08:57:44.6132201Z * [new tag] ciflow/h100/168096 -> ciflow/h100/168096 2025-12-04T08:57:44.6132895Z * [new tag] ciflow/h100/168175 -> ciflow/h100/168175 2025-12-04T08:57:44.6133628Z * [new tag] ciflow/h100/168195 -> ciflow/h100/168195 2025-12-04T08:57:44.6134457Z * [new tag] ciflow/h100/168980 -> ciflow/h100/168980 2025-12-04T08:57:44.6135508Z * [new tag] ciflow/h100/169200 -> ciflow/h100/169200 2025-12-04T08:57:44.6136609Z * [new tag] ciflow/h100/169216 -> ciflow/h100/169216 2025-12-04T08:57:44.6137556Z * [new tag] ciflow/h100/169380 -> ciflow/h100/169380 2025-12-04T08:57:44.6138257Z * [new tag] ciflow/h100/169412 -> ciflow/h100/169412 2025-12-04T08:57:44.6138980Z * [new tag] ciflow/h100/169470 -> ciflow/h100/169470 2025-12-04T08:57:44.6139736Z * [new tag] ciflow/h100/169471 -> ciflow/h100/169471 2025-12-04T08:57:44.6140443Z * [new tag] ciflow/h100/169472 -> ciflow/h100/169472 2025-12-04T08:57:44.6141163Z * [new tag] ciflow/h100/169514 -> ciflow/h100/169514 2025-12-04T08:57:44.6142141Z * [new tag] ciflow/inductor-cu126/168096 -> ciflow/inductor-cu126/168096 2025-12-04T08:57:44.6143332Z * [new tag] ciflow/inductor-micro-benchmark-cpu-x86/168096 -> ciflow/inductor-micro-benchmark-cpu-x86/168096 2025-12-04T08:57:44.6144055Z * [new tag] ciflow/inductor-micro-benchmark/166165 -> ciflow/inductor-micro-benchmark/166165 2025-12-04T08:57:44.6144742Z * [new tag] ciflow/inductor-micro-benchmark/168096 -> ciflow/inductor-micro-benchmark/168096 2025-12-04T08:57:44.6145619Z * [new tag] ciflow/inductor-perf-compare/168096 -> ciflow/inductor-perf-compare/168096 2025-12-04T08:57:44.6146978Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168073 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168073 2025-12-04T08:57:44.6147937Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/168096 -> ciflow/inductor-perf-test-nightly-rocm-mi300/168096 2025-12-04T08:57:44.6148849Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi300/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi300/169024 2025-12-04T08:57:44.6149818Z * [new tag] ciflow/inductor-perf-test-nightly-rocm-mi355/169024 -> ciflow/inductor-perf-test-nightly-rocm-mi355/169024 2025-12-04T08:57:44.6150588Z * [new tag] ciflow/inductor-perf-test-nightly/168096 -> ciflow/inductor-perf-test-nightly/168096 2025-12-04T08:57:44.6151375Z * [new tag] ciflow/inductor-periodic/168096 -> ciflow/inductor-periodic/168096 2025-12-04T08:57:44.6152092Z * [new tag] ciflow/inductor-periodic/169024 -> ciflow/inductor-periodic/169024 2025-12-04T08:57:44.6152927Z * [new tag] ciflow/inductor-periodic/169425 -> ciflow/inductor-periodic/169425 2025-12-04T08:57:44.6154463Z * [new tag] ciflow/inductor-rocm-mi200/165545 -> ciflow/inductor-rocm-mi200/165545 2025-12-04T08:57:44.6155318Z * [new tag] ciflow/inductor-rocm-mi200/165997 -> ciflow/inductor-rocm-mi200/165997 2025-12-04T08:57:44.6156034Z * [new tag] ciflow/inductor-rocm-mi200/168096 -> ciflow/inductor-rocm-mi200/168096 2025-12-04T08:57:44.6156901Z * [new tag] ciflow/inductor-rocm-mi200/169063 -> ciflow/inductor-rocm-mi200/169063 2025-12-04T08:57:44.6157613Z * [new tag] ciflow/inductor-rocm-mi200/169425 -> ciflow/inductor-rocm-mi200/169425 2025-12-04T08:57:44.6158496Z * [new tag] ciflow/inductor-rocm-mi300/165545 -> ciflow/inductor-rocm-mi300/165545 2025-12-04T08:57:44.6159275Z * [new tag] ciflow/inductor-rocm-mi300/168096 -> ciflow/inductor-rocm-mi300/168096 2025-12-04T08:57:44.6160049Z * [new tag] ciflow/inductor-rocm-mi300/169063 -> ciflow/inductor-rocm-mi300/169063 2025-12-04T08:57:44.6160761Z * [new tag] ciflow/inductor-rocm-mi300/169425 -> ciflow/inductor-rocm-mi300/169425 2025-12-04T08:57:44.6161712Z * [new tag] ciflow/inductor-rocm/162052 -> ciflow/inductor-rocm/162052 2025-12-04T08:57:44.6162355Z * [new tag] ciflow/inductor-rocm/168971 -> ciflow/inductor-rocm/168971 2025-12-04T08:57:44.6163223Z * [new tag] ciflow/inductor-windows/168096 -> ciflow/inductor-windows/168096 2025-12-04T08:57:44.6164468Z * [new tag] ciflow/inductor/144542 -> ciflow/inductor/144542 2025-12-04T08:57:44.6165112Z * [new tag] ciflow/inductor/146506 -> ciflow/inductor/146506 2025-12-04T08:57:44.6165839Z * [new tag] ciflow/inductor/147990 -> ciflow/inductor/147990 2025-12-04T08:57:44.6166647Z * [new tag] ciflow/inductor/148294 -> ciflow/inductor/148294 2025-12-04T08:57:44.6167336Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-12-04T08:57:44.6168053Z * [new tag] ciflow/inductor/157149 -> ciflow/inductor/157149 2025-12-04T08:57:44.6168751Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-12-04T08:57:44.6169569Z * [new tag] ciflow/inductor/160685 -> ciflow/inductor/160685 2025-12-04T08:57:44.6170266Z * [new tag] ciflow/inductor/160686 -> ciflow/inductor/160686 2025-12-04T08:57:44.6170974Z * [new tag] ciflow/inductor/160687 -> ciflow/inductor/160687 2025-12-04T08:57:44.6171688Z * [new tag] ciflow/inductor/160688 -> ciflow/inductor/160688 2025-12-04T08:57:44.6172770Z * [new tag] ciflow/inductor/160706 -> ciflow/inductor/160706 2025-12-04T08:57:44.6173766Z * [new tag] ciflow/inductor/160729 -> ciflow/inductor/160729 2025-12-04T08:57:44.6174678Z * [new tag] ciflow/inductor/161938 -> ciflow/inductor/161938 2025-12-04T08:57:44.6175370Z * [new tag] ciflow/inductor/161939 -> ciflow/inductor/161939 2025-12-04T08:57:44.6176109Z * [new tag] ciflow/inductor/161940 -> ciflow/inductor/161940 2025-12-04T08:57:44.6176880Z * [new tag] ciflow/inductor/162052 -> ciflow/inductor/162052 2025-12-04T08:57:44.6177572Z * [new tag] ciflow/inductor/162275 -> ciflow/inductor/162275 2025-12-04T08:57:44.6178311Z * [new tag] ciflow/inductor/162795 -> ciflow/inductor/162795 2025-12-04T08:57:44.6179307Z * [new tag] ciflow/inductor/163245 -> ciflow/inductor/163245 2025-12-04T08:57:44.6179998Z * [new tag] ciflow/inductor/163335 -> ciflow/inductor/163335 2025-12-04T08:57:44.6180726Z * [new tag] ciflow/inductor/163503 -> ciflow/inductor/163503 2025-12-04T08:57:44.6181435Z * [new tag] ciflow/inductor/163942 -> ciflow/inductor/163942 2025-12-04T08:57:44.6182364Z * [new tag] ciflow/inductor/165270 -> ciflow/inductor/165270 2025-12-04T08:57:44.6183059Z * [new tag] ciflow/inductor/165274 -> ciflow/inductor/165274 2025-12-04T08:57:44.6183783Z * [new tag] ciflow/inductor/165322 -> ciflow/inductor/165322 2025-12-04T08:57:44.6184524Z * [new tag] ciflow/inductor/165597 -> ciflow/inductor/165597 2025-12-04T08:57:44.6185230Z * [new tag] ciflow/inductor/166063 -> ciflow/inductor/166063 2025-12-04T08:57:44.6185980Z * [new tag] ciflow/inductor/166075 -> ciflow/inductor/166075 2025-12-04T08:57:44.6186702Z * [new tag] ciflow/inductor/166165 -> ciflow/inductor/166165 2025-12-04T08:57:44.6188145Z * [new tag] ciflow/inductor/166254 -> ciflow/inductor/166254 2025-12-04T08:57:44.6188837Z * [new tag] ciflow/inductor/166483 -> ciflow/inductor/166483 2025-12-04T08:57:44.6189652Z * [new tag] ciflow/inductor/166494 -> ciflow/inductor/166494 2025-12-04T08:57:44.6190384Z * [new tag] ciflow/inductor/166545 -> ciflow/inductor/166545 2025-12-04T08:57:44.6191127Z * [new tag] ciflow/inductor/166788 -> ciflow/inductor/166788 2025-12-04T08:57:44.6192101Z * [new tag] ciflow/inductor/166846 -> ciflow/inductor/166846 2025-12-04T08:57:44.6192814Z * [new tag] ciflow/inductor/167300 -> ciflow/inductor/167300 2025-12-04T08:57:44.6193554Z * [new tag] ciflow/inductor/167407 -> ciflow/inductor/167407 2025-12-04T08:57:44.6194494Z * [new tag] ciflow/inductor/167536 -> ciflow/inductor/167536 2025-12-04T08:57:44.6195284Z * [new tag] ciflow/inductor/167552 -> ciflow/inductor/167552 2025-12-04T08:57:44.6196075Z * [new tag] ciflow/inductor/167555 -> ciflow/inductor/167555 2025-12-04T08:57:44.6196986Z * [new tag] ciflow/inductor/167583 -> ciflow/inductor/167583 2025-12-04T08:57:44.6197688Z * [new tag] ciflow/inductor/167599 -> ciflow/inductor/167599 2025-12-04T08:57:44.6198431Z * [new tag] ciflow/inductor/167647 -> ciflow/inductor/167647 2025-12-04T08:57:44.6199178Z * [new tag] ciflow/inductor/167677 -> ciflow/inductor/167677 2025-12-04T08:57:44.6200053Z * [new tag] ciflow/inductor/167680 -> ciflow/inductor/167680 2025-12-04T08:57:44.6200779Z * [new tag] ciflow/inductor/167695 -> ciflow/inductor/167695 2025-12-04T08:57:44.6201510Z * [new tag] ciflow/inductor/167742 -> ciflow/inductor/167742 2025-12-04T08:57:44.6202245Z * [new tag] ciflow/inductor/167768 -> ciflow/inductor/167768 2025-12-04T08:57:44.6203254Z * [new tag] ciflow/inductor/167773 -> ciflow/inductor/167773 2025-12-04T08:57:44.6203927Z * [new tag] ciflow/inductor/167781 -> ciflow/inductor/167781 2025-12-04T08:57:44.6204694Z * [new tag] ciflow/inductor/167880 -> ciflow/inductor/167880 2025-12-04T08:57:44.6205454Z * [new tag] ciflow/inductor/167887 -> ciflow/inductor/167887 2025-12-04T08:57:44.6206167Z * [new tag] ciflow/inductor/167972 -> ciflow/inductor/167972 2025-12-04T08:57:44.6206900Z * [new tag] ciflow/inductor/167989 -> ciflow/inductor/167989 2025-12-04T08:57:44.6207622Z * [new tag] ciflow/inductor/168002 -> ciflow/inductor/168002 2025-12-04T08:57:44.6208349Z * [new tag] ciflow/inductor/168050 -> ciflow/inductor/168050 2025-12-04T08:57:44.6209095Z * [new tag] ciflow/inductor/168051 -> ciflow/inductor/168051 2025-12-04T08:57:44.6209822Z * [new tag] ciflow/inductor/168052 -> ciflow/inductor/168052 2025-12-04T08:57:44.6210603Z * [new tag] ciflow/inductor/168073 -> ciflow/inductor/168073 2025-12-04T08:57:44.6211293Z * [new tag] ciflow/inductor/168096 -> ciflow/inductor/168096 2025-12-04T08:57:44.6212051Z * [new tag] ciflow/inductor/168114 -> ciflow/inductor/168114 2025-12-04T08:57:44.6212768Z * [new tag] ciflow/inductor/168115 -> ciflow/inductor/168115 2025-12-04T08:57:44.6214021Z * [new tag] ciflow/inductor/168127 -> ciflow/inductor/168127 2025-12-04T08:57:44.6214711Z * [new tag] ciflow/inductor/168129 -> ciflow/inductor/168129 2025-12-04T08:57:44.6215434Z * [new tag] ciflow/inductor/168157 -> ciflow/inductor/168157 2025-12-04T08:57:44.6216348Z * [new tag] ciflow/inductor/168175 -> ciflow/inductor/168175 2025-12-04T08:57:44.6217122Z * [new tag] ciflow/inductor/168185 -> ciflow/inductor/168185 2025-12-04T08:57:44.6217778Z * [new tag] ciflow/inductor/168195 -> ciflow/inductor/168195 2025-12-04T08:57:44.6218501Z * [new tag] ciflow/inductor/168209 -> ciflow/inductor/168209 2025-12-04T08:57:44.6219251Z * [new tag] ciflow/inductor/168266 -> ciflow/inductor/168266 2025-12-04T08:57:44.6220246Z * [new tag] ciflow/inductor/168316 -> ciflow/inductor/168316 2025-12-04T08:57:44.6221064Z * [new tag] ciflow/inductor/168326 -> ciflow/inductor/168326 2025-12-04T08:57:44.6221835Z * [new tag] ciflow/inductor/168368 -> ciflow/inductor/168368 2025-12-04T08:57:44.6222572Z * [new tag] ciflow/inductor/168894 -> ciflow/inductor/168894 2025-12-04T08:57:44.6223346Z * [new tag] ciflow/inductor/168934 -> ciflow/inductor/168934 2025-12-04T08:57:44.6224295Z * [new tag] ciflow/inductor/168939 -> ciflow/inductor/168939 2025-12-04T08:57:44.6225259Z * [new tag] ciflow/inductor/168946 -> ciflow/inductor/168946 2025-12-04T08:57:44.6226002Z * [new tag] ciflow/inductor/168950 -> ciflow/inductor/168950 2025-12-04T08:57:44.6226779Z * [new tag] ciflow/inductor/168951 -> ciflow/inductor/168951 2025-12-04T08:57:44.6227596Z * [new tag] ciflow/inductor/168952 -> ciflow/inductor/168952 2025-12-04T08:57:44.6228416Z * [new tag] ciflow/inductor/168955 -> ciflow/inductor/168955 2025-12-04T08:57:44.6229180Z * [new tag] ciflow/inductor/168971 -> ciflow/inductor/168971 2025-12-04T08:57:44.6229927Z * [new tag] ciflow/inductor/168979 -> ciflow/inductor/168979 2025-12-04T08:57:44.6230693Z * [new tag] ciflow/inductor/168980 -> ciflow/inductor/168980 2025-12-04T08:57:44.6231670Z * [new tag] ciflow/inductor/168983 -> ciflow/inductor/168983 2025-12-04T08:57:44.6232357Z * [new tag] ciflow/inductor/169006 -> ciflow/inductor/169006 2025-12-04T08:57:44.6233119Z * [new tag] ciflow/inductor/169023 -> ciflow/inductor/169023 2025-12-04T08:57:44.6233876Z * [new tag] ciflow/inductor/169024 -> ciflow/inductor/169024 2025-12-04T08:57:44.6234684Z * [new tag] ciflow/inductor/169025 -> ciflow/inductor/169025 2025-12-04T08:57:44.6235434Z * [new tag] ciflow/inductor/169066 -> ciflow/inductor/169066 2025-12-04T08:57:44.6236200Z * [new tag] ciflow/inductor/169091 -> ciflow/inductor/169091 2025-12-04T08:57:44.6236963Z * [new tag] ciflow/inductor/169102 -> ciflow/inductor/169102 2025-12-04T08:57:44.6237731Z * [new tag] ciflow/inductor/169103 -> ciflow/inductor/169103 2025-12-04T08:57:44.6238621Z * [new tag] ciflow/inductor/169121 -> ciflow/inductor/169121 2025-12-04T08:57:44.6239386Z * [new tag] ciflow/inductor/169134 -> ciflow/inductor/169134 2025-12-04T08:57:44.6240116Z * [new tag] ciflow/inductor/169135 -> ciflow/inductor/169135 2025-12-04T08:57:44.6240859Z * [new tag] ciflow/inductor/169141 -> ciflow/inductor/169141 2025-12-04T08:57:44.6241591Z * [new tag] ciflow/inductor/169151 -> ciflow/inductor/169151 2025-12-04T08:57:44.6242523Z * [new tag] ciflow/inductor/169161 -> ciflow/inductor/169161 2025-12-04T08:57:44.6243187Z * [new tag] ciflow/inductor/169167 -> ciflow/inductor/169167 2025-12-04T08:57:44.6244164Z * [new tag] ciflow/inductor/169177 -> ciflow/inductor/169177 2025-12-04T08:57:44.6244948Z * [new tag] ciflow/inductor/169185 -> ciflow/inductor/169185 2025-12-04T08:57:44.6245876Z * [new tag] ciflow/inductor/169196 -> ciflow/inductor/169196 2025-12-04T08:57:44.6246704Z * [new tag] ciflow/inductor/169200 -> ciflow/inductor/169200 2025-12-04T08:57:44.6247334Z * [new tag] ciflow/inductor/169204 -> ciflow/inductor/169204 2025-12-04T08:57:44.6248069Z * [new tag] ciflow/inductor/169216 -> ciflow/inductor/169216 2025-12-04T08:57:44.6248792Z * [new tag] ciflow/inductor/169219 -> ciflow/inductor/169219 2025-12-04T08:57:44.6249530Z * [new tag] ciflow/inductor/169220 -> ciflow/inductor/169220 2025-12-04T08:57:44.6250474Z * [new tag] ciflow/inductor/169230 -> ciflow/inductor/169230 2025-12-04T08:57:44.6251163Z * [new tag] ciflow/inductor/169242 -> ciflow/inductor/169242 2025-12-04T08:57:44.6251916Z * [new tag] ciflow/inductor/169245 -> ciflow/inductor/169245 2025-12-04T08:57:44.6252833Z * [new tag] ciflow/inductor/169260 -> ciflow/inductor/169260 2025-12-04T08:57:44.6253536Z * [new tag] ciflow/inductor/169282 -> ciflow/inductor/169282 2025-12-04T08:57:44.6254275Z * [new tag] ciflow/inductor/169286 -> ciflow/inductor/169286 2025-12-04T08:57:44.6255079Z * [new tag] ciflow/inductor/169299 -> ciflow/inductor/169299 2025-12-04T08:57:44.6255913Z * [new tag] ciflow/inductor/169304 -> ciflow/inductor/169304 2025-12-04T08:57:44.6257152Z * [new tag] ciflow/inductor/169305 -> ciflow/inductor/169305 2025-12-04T08:57:44.6257860Z * [new tag] ciflow/inductor/169308 -> ciflow/inductor/169308 2025-12-04T08:57:44.6258605Z * [new tag] ciflow/inductor/169319 -> ciflow/inductor/169319 2025-12-04T08:57:44.6259322Z * [new tag] ciflow/inductor/169326 -> ciflow/inductor/169326 2025-12-04T08:57:44.6260070Z * [new tag] ciflow/inductor/169332 -> ciflow/inductor/169332 2025-12-04T08:57:44.6260834Z * [new tag] ciflow/inductor/169333 -> ciflow/inductor/169333 2025-12-04T08:57:44.6261774Z * [new tag] ciflow/inductor/169336 -> ciflow/inductor/169336 2025-12-04T08:57:44.6262514Z * [new tag] ciflow/inductor/169340 -> ciflow/inductor/169340 2025-12-04T08:57:44.6263277Z * [new tag] ciflow/inductor/169341 -> ciflow/inductor/169341 2025-12-04T08:57:44.6264119Z * [new tag] ciflow/inductor/169343 -> ciflow/inductor/169343 2025-12-04T08:57:44.6265273Z * [new tag] ciflow/inductor/169346 -> ciflow/inductor/169346 2025-12-04T08:57:44.6266125Z * [new tag] ciflow/inductor/169348 -> ciflow/inductor/169348 2025-12-04T08:57:44.6266985Z * [new tag] ciflow/inductor/169350 -> ciflow/inductor/169350 2025-12-04T08:57:44.6268155Z * [new tag] ciflow/inductor/169355 -> ciflow/inductor/169355 2025-12-04T08:57:44.6268939Z * [new tag] ciflow/inductor/169370 -> ciflow/inductor/169370 2025-12-04T08:57:44.6270024Z * [new tag] ciflow/inductor/169375 -> ciflow/inductor/169375 2025-12-04T08:57:44.6270777Z * [new tag] ciflow/inductor/169389 -> ciflow/inductor/169389 2025-12-04T08:57:44.6271566Z * [new tag] ciflow/inductor/169391 -> ciflow/inductor/169391 2025-12-04T08:57:44.6272417Z * [new tag] ciflow/inductor/169393 -> ciflow/inductor/169393 2025-12-04T08:57:44.6273213Z * [new tag] ciflow/inductor/169399 -> ciflow/inductor/169399 2025-12-04T08:57:44.6274091Z * [new tag] ciflow/inductor/169400 -> ciflow/inductor/169400 2025-12-04T08:57:44.6274889Z * [new tag] ciflow/inductor/169415 -> ciflow/inductor/169415 2025-12-04T08:57:44.6275647Z * [new tag] ciflow/inductor/169417 -> ciflow/inductor/169417 2025-12-04T08:57:44.6276502Z * [new tag] ciflow/inductor/169418 -> ciflow/inductor/169418 2025-12-04T08:57:44.6277458Z * [new tag] ciflow/inductor/169430 -> ciflow/inductor/169430 2025-12-04T08:57:44.6278292Z * [new tag] ciflow/inductor/169432 -> ciflow/inductor/169432 2025-12-04T08:57:44.6278980Z * [new tag] ciflow/inductor/169436 -> ciflow/inductor/169436 2025-12-04T08:57:44.6279997Z * [new tag] ciflow/inductor/169437 -> ciflow/inductor/169437 2025-12-04T08:57:44.6280721Z * [new tag] ciflow/inductor/169438 -> ciflow/inductor/169438 2025-12-04T08:57:44.6281471Z * [new tag] ciflow/inductor/169441 -> ciflow/inductor/169441 2025-12-04T08:57:44.6282214Z * [new tag] ciflow/inductor/169446 -> ciflow/inductor/169446 2025-12-04T08:57:44.6283105Z * [new tag] ciflow/inductor/169447 -> ciflow/inductor/169447 2025-12-04T08:57:44.6283847Z * [new tag] ciflow/inductor/169452 -> ciflow/inductor/169452 2025-12-04T08:57:44.6284729Z * [new tag] ciflow/inductor/169455 -> ciflow/inductor/169455 2025-12-04T08:57:44.6285448Z * [new tag] ciflow/inductor/169459 -> ciflow/inductor/169459 2025-12-04T08:57:44.6286335Z * [new tag] ciflow/inductor/169463 -> ciflow/inductor/169463 2025-12-04T08:57:44.6287197Z * [new tag] ciflow/inductor/169476 -> ciflow/inductor/169476 2025-12-04T08:57:44.6287938Z * [new tag] ciflow/inductor/169485 -> ciflow/inductor/169485 2025-12-04T08:57:44.6288675Z * [new tag] ciflow/inductor/169493 -> ciflow/inductor/169493 2025-12-04T08:57:44.6289421Z * [new tag] ciflow/inductor/169496 -> ciflow/inductor/169496 2025-12-04T08:57:44.6290224Z * [new tag] ciflow/inductor/169497 -> ciflow/inductor/169497 2025-12-04T08:57:44.6290955Z * [new tag] ciflow/inductor/169503 -> ciflow/inductor/169503 2025-12-04T08:57:44.6291701Z * [new tag] ciflow/inductor/169504 -> ciflow/inductor/169504 2025-12-04T08:57:44.6292818Z * [new tag] ciflow/inductor/169505 -> ciflow/inductor/169505 2025-12-04T08:57:44.6294084Z * [new tag] ciflow/inductor/169508 -> ciflow/inductor/169508 2025-12-04T08:57:44.6294749Z * [new tag] ciflow/inductor/169509 -> ciflow/inductor/169509 2025-12-04T08:57:44.6295501Z * [new tag] ciflow/inductor/169513 -> ciflow/inductor/169513 2025-12-04T08:57:44.6296278Z * [new tag] ciflow/inductor/169514 -> ciflow/inductor/169514 2025-12-04T08:57:44.6297029Z * [new tag] ciflow/inductor/169515 -> ciflow/inductor/169515 2025-12-04T08:57:44.6297998Z * [new tag] ciflow/inductor/169517 -> ciflow/inductor/169517 2025-12-04T08:57:44.6298845Z * [new tag] ciflow/inductor/169519 -> ciflow/inductor/169519 2025-12-04T08:57:44.6299625Z * [new tag] ciflow/inductor/169520 -> ciflow/inductor/169520 2025-12-04T08:57:44.6300378Z * [new tag] ciflow/inductor/169521 -> ciflow/inductor/169521 2025-12-04T08:57:44.6301128Z * [new tag] ciflow/inductor/169524 -> ciflow/inductor/169524 2025-12-04T08:57:44.6301869Z * [new tag] ciflow/inductor/169527 -> ciflow/inductor/169527 2025-12-04T08:57:44.6302626Z * [new tag] ciflow/inductor/169528 -> ciflow/inductor/169528 2025-12-04T08:57:44.6303500Z * [new tag] ciflow/inductor/169532 -> ciflow/inductor/169532 2025-12-04T08:57:44.6304262Z * [new tag] ciflow/inductor/169535 -> ciflow/inductor/169535 2025-12-04T08:57:44.6304996Z * [new tag] ciflow/inductor/169536 -> ciflow/inductor/169536 2025-12-04T08:57:44.6305743Z * [new tag] ciflow/inductor/169547 -> ciflow/inductor/169547 2025-12-04T08:57:44.6306562Z * [new tag] ciflow/inductor/169548 -> ciflow/inductor/169548 2025-12-04T08:57:44.6307357Z * [new tag] ciflow/inductor/169549 -> ciflow/inductor/169549 2025-12-04T08:57:44.6308358Z * [new tag] ciflow/inductor/169551 -> ciflow/inductor/169551 2025-12-04T08:57:44.6309145Z * [new tag] ciflow/inductor/169552 -> ciflow/inductor/169552 2025-12-04T08:57:44.6309926Z * [new tag] ciflow/inductor/169553 -> ciflow/inductor/169553 2025-12-04T08:57:44.6310970Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-12-04T08:57:44.6311972Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-12-04T08:57:44.6312893Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-12-04T08:57:44.6313863Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-12-04T08:57:44.6315036Z * [new tag] ciflow/linux-aarch64/166075 -> ciflow/linux-aarch64/166075 2025-12-04T08:57:44.6333570Z * [new tag] ciflow/linux-aarch64/166876 -> ciflow/linux-aarch64/166876 2025-12-04T08:57:44.6334236Z * [new tag] ciflow/linux-aarch64/167981 -> ciflow/linux-aarch64/167981 2025-12-04T08:57:44.6335521Z * [new tag] ciflow/mps/166254 -> ciflow/mps/166254 2025-12-04T08:57:44.6336214Z * [new tag] ciflow/mps/169017 -> ciflow/mps/169017 2025-12-04T08:57:44.6336711Z * [new tag] ciflow/mps/169372 -> ciflow/mps/169372 2025-12-04T08:57:44.6337214Z * [new tag] ciflow/mps/169478 -> ciflow/mps/169478 2025-12-04T08:57:44.6337777Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-12-04T08:57:44.6338371Z * [new tag] ciflow/op-benchmark/166075 -> ciflow/op-benchmark/166075 2025-12-04T08:57:44.6338958Z * [new tag] ciflow/op-benchmark/169544 -> ciflow/op-benchmark/169544 2025-12-04T08:57:44.6339638Z * [new tag] ciflow/periodic-rocm-mi200/165997 -> ciflow/periodic-rocm-mi200/165997 2025-12-04T08:57:44.6340366Z * [new tag] ciflow/periodic-rocm-mi200/166517 -> ciflow/periodic-rocm-mi200/166517 2025-12-04T08:57:44.6341084Z * [new tag] ciflow/periodic-rocm-mi200/169063 -> ciflow/periodic-rocm-mi200/169063 2025-12-04T08:57:44.6341793Z * [new tag] ciflow/periodic-rocm-mi200/169425 -> ciflow/periodic-rocm-mi200/169425 2025-12-04T08:57:44.6342516Z * [new tag] ciflow/periodic-rocm-mi300/166517 -> ciflow/periodic-rocm-mi300/166517 2025-12-04T08:57:44.6343234Z * [new tag] ciflow/periodic-rocm-mi300/169063 -> ciflow/periodic-rocm-mi300/169063 2025-12-04T08:57:44.6343953Z * [new tag] ciflow/periodic-rocm-mi300/169425 -> ciflow/periodic-rocm-mi300/169425 2025-12-04T08:57:44.6344596Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-12-04T08:57:44.6345170Z * [new tag] ciflow/periodic/167207 -> ciflow/periodic/167207 2025-12-04T08:57:44.6345725Z * [new tag] ciflow/periodic/167978 -> ciflow/periodic/167978 2025-12-04T08:57:44.6346263Z * [new tag] ciflow/periodic/168096 -> ciflow/periodic/168096 2025-12-04T08:57:44.6346804Z * [new tag] ciflow/periodic/169286 -> ciflow/periodic/169286 2025-12-04T08:57:44.6347450Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-12-04T08:57:44.6348198Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-12-04T08:57:44.6348759Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-12-04T08:57:44.6349327Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-12-04T08:57:44.6350450Z * [new tag] ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> ciflow/periodic/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T08:57:44.6351491Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-12-04T08:57:44.6352144Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-12-04T08:57:44.6352813Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-12-04T08:57:44.6353474Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-12-04T08:57:44.6354124Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-12-04T08:57:44.6354786Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-12-04T08:57:44.6355441Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-12-04T08:57:44.6356022Z * [new tag] ciflow/pull/167207 -> ciflow/pull/167207 2025-12-04T08:57:44.6356698Z * [new tag] ciflow/quantization-periodic/169207 -> ciflow/quantization-periodic/169207 2025-12-04T08:57:44.6357385Z * [new tag] ciflow/rocm-mi200/165545 -> ciflow/rocm-mi200/165545 2025-12-04T08:57:44.6357957Z * [new tag] ciflow/rocm-mi200/165997 -> ciflow/rocm-mi200/165997 2025-12-04T08:57:44.6358529Z * [new tag] ciflow/rocm-mi200/168096 -> ciflow/rocm-mi200/168096 2025-12-04T08:57:44.6359094Z * [new tag] ciflow/rocm-mi200/168275 -> ciflow/rocm-mi200/168275 2025-12-04T08:57:44.6359764Z * [new tag] ciflow/rocm-mi200/169063 -> ciflow/rocm-mi200/169063 2025-12-04T08:57:44.6360318Z * [new tag] ciflow/rocm-mi200/169356 -> ciflow/rocm-mi200/169356 2025-12-04T08:57:44.6360865Z * [new tag] ciflow/rocm-mi200/169425 -> ciflow/rocm-mi200/169425 2025-12-04T08:57:44.6361405Z * [new tag] ciflow/rocm-mi300/165545 -> ciflow/rocm-mi300/165545 2025-12-04T08:57:44.6361955Z * [new tag] ciflow/rocm-mi300/167157 -> ciflow/rocm-mi300/167157 2025-12-04T08:57:44.6362499Z * [new tag] ciflow/rocm-mi300/168096 -> ciflow/rocm-mi300/168096 2025-12-04T08:57:44.6363048Z * [new tag] ciflow/rocm-mi300/169063 -> ciflow/rocm-mi300/169063 2025-12-04T08:57:44.6363584Z * [new tag] ciflow/rocm-mi300/169425 -> ciflow/rocm-mi300/169425 2025-12-04T08:57:44.6364133Z * [new tag] ciflow/rocm-mi355/167157 -> ciflow/rocm-mi355/167157 2025-12-04T08:57:44.6364680Z * [new tag] ciflow/rocm-mi355/168275 -> ciflow/rocm-mi355/168275 2025-12-04T08:57:44.6365215Z * [new tag] ciflow/rocm-mi355/169425 -> ciflow/rocm-mi355/169425 2025-12-04T08:57:44.6365780Z * [new tag] ciflow/rocm-navi31/168275 -> ciflow/rocm-navi31/168275 2025-12-04T08:57:44.6366353Z * [new tag] ciflow/rocm-navi31/169425 -> ciflow/rocm-navi31/169425 2025-12-04T08:57:44.6366898Z * [new tag] ciflow/rocm/115316 -> ciflow/rocm/115316 2025-12-04T08:57:44.6367393Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-12-04T08:57:44.6367891Z * [new tag] ciflow/rocm/160685 -> ciflow/rocm/160685 2025-12-04T08:57:44.6368391Z * [new tag] ciflow/rocm/161607 -> ciflow/rocm/161607 2025-12-04T08:57:44.6368890Z * [new tag] ciflow/rocm/162052 -> ciflow/rocm/162052 2025-12-04T08:57:44.6369373Z * [new tag] ciflow/rocm/165997 -> ciflow/rocm/165997 2025-12-04T08:57:44.6369873Z * [new tag] ciflow/rocm/166165 -> ciflow/rocm/166165 2025-12-04T08:57:44.6370369Z * [new tag] ciflow/rocm/166517 -> ciflow/rocm/166517 2025-12-04T08:57:44.6370869Z * [new tag] ciflow/rocm/167207 -> ciflow/rocm/167207 2025-12-04T08:57:44.6371455Z * [new tag] ciflow/rocm/167536 -> ciflow/rocm/167536 2025-12-04T08:57:44.6372013Z * [new tag] ciflow/rocm/167781 -> ciflow/rocm/167781 2025-12-04T08:57:44.6372514Z * [new tag] ciflow/rocm/167989 -> ciflow/rocm/167989 2025-12-04T08:57:44.6373017Z * [new tag] ciflow/rocm/168073 -> ciflow/rocm/168073 2025-12-04T08:57:44.6373506Z * [new tag] ciflow/rocm/168195 -> ciflow/rocm/168195 2025-12-04T08:57:44.6374002Z * [new tag] ciflow/rocm/168939 -> ciflow/rocm/168939 2025-12-04T08:57:44.6374503Z * [new tag] ciflow/rocm/168971 -> ciflow/rocm/168971 2025-12-04T08:57:44.6374989Z * [new tag] ciflow/rocm/169024 -> ciflow/rocm/169024 2025-12-04T08:57:44.6375488Z * [new tag] ciflow/rocm/169200 -> ciflow/rocm/169200 2025-12-04T08:57:44.6375999Z * [new tag] ciflow/rocm/169216 -> ciflow/rocm/169216 2025-12-04T08:57:44.6376505Z * [new tag] ciflow/rocm/169312 -> ciflow/rocm/169312 2025-12-04T08:57:44.6377127Z * [new tag] ciflow/rocm/169380 -> ciflow/rocm/169380 2025-12-04T08:57:44.6377869Z * [new tag] ciflow/rocm/169427 -> ciflow/rocm/169427 2025-12-04T08:57:44.6378589Z * [new tag] ciflow/rocm/169455 -> ciflow/rocm/169455 2025-12-04T08:57:44.6379335Z * [new tag] ciflow/rocm/169470 -> ciflow/rocm/169470 2025-12-04T08:57:44.6380065Z * [new tag] ciflow/rocm/169471 -> ciflow/rocm/169471 2025-12-04T08:57:44.6380813Z * [new tag] ciflow/rocm/169472 -> ciflow/rocm/169472 2025-12-04T08:57:44.6381541Z * [new tag] ciflow/rocm/169514 -> ciflow/rocm/169514 2025-12-04T08:57:44.6382620Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-12-04T08:57:44.6383752Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-12-04T08:57:44.6385132Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-12-04T08:57:44.6386036Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-12-04T08:57:44.6386546Z * [new tag] ciflow/slow/167207 -> ciflow/slow/167207 2025-12-04T08:57:44.6387064Z * [new tag] ciflow/slow/168050 -> ciflow/slow/168050 2025-12-04T08:57:44.6388383Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-12-04T08:57:44.6389325Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-12-04T08:57:44.6390616Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-12-04T08:57:44.6391736Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-12-04T08:57:44.6392819Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-12-04T08:57:44.6393804Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-12-04T08:57:44.6394750Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-12-04T08:57:44.6395647Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-12-04T08:57:44.6397370Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-12-04T08:57:44.6398256Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-12-04T08:57:44.6398791Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-12-04T08:57:44.6399779Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-12-04T08:57:44.6400796Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-12-04T08:57:44.6401674Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-12-04T08:57:44.6403121Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-12-04T08:57:44.6404049Z * [new tag] ciflow/torchbench/168175 -> ciflow/torchbench/168175 2025-12-04T08:57:44.6404598Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-12-04T08:57:44.6405192Z * [new tag] ciflow/trunk/157149 -> ciflow/trunk/157149 2025-12-04T08:57:44.6405911Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-12-04T08:57:44.6406642Z * [new tag] ciflow/trunk/159718 -> ciflow/trunk/159718 2025-12-04T08:57:44.6407314Z * [new tag] ciflow/trunk/160685 -> ciflow/trunk/160685 2025-12-04T08:57:44.6408019Z * [new tag] ciflow/trunk/160729 -> ciflow/trunk/160729 2025-12-04T08:57:44.6408719Z * [new tag] ciflow/trunk/162275 -> ciflow/trunk/162275 2025-12-04T08:57:44.6409469Z * [new tag] ciflow/trunk/162795 -> ciflow/trunk/162795 2025-12-04T08:57:44.6410155Z * [new tag] ciflow/trunk/163245 -> ciflow/trunk/163245 2025-12-04T08:57:44.6410837Z * [new tag] ciflow/trunk/163942 -> ciflow/trunk/163942 2025-12-04T08:57:44.6411515Z * [new tag] ciflow/trunk/165274 -> ciflow/trunk/165274 2025-12-04T08:57:44.6412623Z * [new tag] ciflow/trunk/165483 -> ciflow/trunk/165483 2025-12-04T08:57:44.6413573Z * [new tag] ciflow/trunk/165728 -> ciflow/trunk/165728 2025-12-04T08:57:44.6414467Z * [new tag] ciflow/trunk/165922 -> ciflow/trunk/165922 2025-12-04T08:57:44.6415223Z * [new tag] ciflow/trunk/166075 -> ciflow/trunk/166075 2025-12-04T08:57:44.6416028Z * [new tag] ciflow/trunk/166165 -> ciflow/trunk/166165 2025-12-04T08:57:44.6416732Z * [new tag] ciflow/trunk/166829 -> ciflow/trunk/166829 2025-12-04T08:57:44.6417626Z * [new tag] ciflow/trunk/166843 -> ciflow/trunk/166843 2025-12-04T08:57:44.6418358Z * [new tag] ciflow/trunk/166876 -> ciflow/trunk/166876 2025-12-04T08:57:44.6419113Z * [new tag] ciflow/trunk/167207 -> ciflow/trunk/167207 2025-12-04T08:57:44.6419837Z * [new tag] ciflow/trunk/167536 -> ciflow/trunk/167536 2025-12-04T08:57:44.6420569Z * [new tag] ciflow/trunk/167552 -> ciflow/trunk/167552 2025-12-04T08:57:44.6421426Z * [new tag] ciflow/trunk/167555 -> ciflow/trunk/167555 2025-12-04T08:57:44.6422180Z * [new tag] ciflow/trunk/167599 -> ciflow/trunk/167599 2025-12-04T08:57:44.6422884Z * [new tag] ciflow/trunk/167659 -> ciflow/trunk/167659 2025-12-04T08:57:44.6424113Z * [new tag] ciflow/trunk/167672 -> ciflow/trunk/167672 2025-12-04T08:57:44.6425003Z * [new tag] ciflow/trunk/167742 -> ciflow/trunk/167742 2025-12-04T08:57:44.6425771Z * [new tag] ciflow/trunk/167781 -> ciflow/trunk/167781 2025-12-04T08:57:44.6426740Z * [new tag] ciflow/trunk/167837 -> ciflow/trunk/167837 2025-12-04T08:57:44.6427604Z * [new tag] ciflow/trunk/167887 -> ciflow/trunk/167887 2025-12-04T08:57:44.6428426Z * [new tag] ciflow/trunk/167978 -> ciflow/trunk/167978 2025-12-04T08:57:44.6429207Z * [new tag] ciflow/trunk/168050 -> ciflow/trunk/168050 2025-12-04T08:57:44.6429946Z * [new tag] ciflow/trunk/168051 -> ciflow/trunk/168051 2025-12-04T08:57:44.6430836Z * [new tag] ciflow/trunk/168096 -> ciflow/trunk/168096 2025-12-04T08:57:44.6431482Z * [new tag] ciflow/trunk/168127 -> ciflow/trunk/168127 2025-12-04T08:57:44.6432264Z * [new tag] ciflow/trunk/168157 -> ciflow/trunk/168157 2025-12-04T08:57:44.6432997Z * [new tag] ciflow/trunk/168175 -> ciflow/trunk/168175 2025-12-04T08:57:44.6433745Z * [new tag] ciflow/trunk/168209 -> ciflow/trunk/168209 2025-12-04T08:57:44.6434695Z * [new tag] ciflow/trunk/168213 -> ciflow/trunk/168213 2025-12-04T08:57:44.6435606Z * [new tag] ciflow/trunk/168226 -> ciflow/trunk/168226 2025-12-04T08:57:44.6436371Z * [new tag] ciflow/trunk/168262 -> ciflow/trunk/168262 2025-12-04T08:57:44.6437125Z * [new tag] ciflow/trunk/168275 -> ciflow/trunk/168275 2025-12-04T08:57:44.6438018Z * [new tag] ciflow/trunk/168328 -> ciflow/trunk/168328 2025-12-04T08:57:44.6438786Z * [new tag] ciflow/trunk/168368 -> ciflow/trunk/168368 2025-12-04T08:57:44.6439611Z * [new tag] ciflow/trunk/168917 -> ciflow/trunk/168917 2025-12-04T08:57:44.6440351Z * [new tag] ciflow/trunk/168933 -> ciflow/trunk/168933 2025-12-04T08:57:44.6441243Z * [new tag] ciflow/trunk/168941 -> ciflow/trunk/168941 2025-12-04T08:57:44.6442033Z * [new tag] ciflow/trunk/168955 -> ciflow/trunk/168955 2025-12-04T08:57:44.6442731Z * [new tag] ciflow/trunk/168980 -> ciflow/trunk/168980 2025-12-04T08:57:44.6443672Z * [new tag] ciflow/trunk/169004 -> ciflow/trunk/169004 2025-12-04T08:57:44.6444418Z * [new tag] ciflow/trunk/169006 -> ciflow/trunk/169006 2025-12-04T08:57:44.6445160Z * [new tag] ciflow/trunk/169023 -> ciflow/trunk/169023 2025-12-04T08:57:44.6445906Z * [new tag] ciflow/trunk/169025 -> ciflow/trunk/169025 2025-12-04T08:57:44.6446620Z * [new tag] ciflow/trunk/169048 -> ciflow/trunk/169048 2025-12-04T08:57:44.6447467Z * [new tag] ciflow/trunk/169066 -> ciflow/trunk/169066 2025-12-04T08:57:44.6448215Z * [new tag] ciflow/trunk/169091 -> ciflow/trunk/169091 2025-12-04T08:57:44.6448944Z * [new tag] ciflow/trunk/169102 -> ciflow/trunk/169102 2025-12-04T08:57:44.6449688Z * [new tag] ciflow/trunk/169103 -> ciflow/trunk/169103 2025-12-04T08:57:44.6450535Z * [new tag] ciflow/trunk/169125 -> ciflow/trunk/169125 2025-12-04T08:57:44.6451403Z * [new tag] ciflow/trunk/169139 -> ciflow/trunk/169139 2025-12-04T08:57:44.6452764Z * [new tag] ciflow/trunk/169148 -> ciflow/trunk/169148 2025-12-04T08:57:44.6453443Z * [new tag] ciflow/trunk/169151 -> ciflow/trunk/169151 2025-12-04T08:57:44.6454172Z * [new tag] ciflow/trunk/169156 -> ciflow/trunk/169156 2025-12-04T08:57:44.6455053Z * [new tag] ciflow/trunk/169176 -> ciflow/trunk/169176 2025-12-04T08:57:44.6455859Z * [new tag] ciflow/trunk/169204 -> ciflow/trunk/169204 2025-12-04T08:57:44.6456631Z * [new tag] ciflow/trunk/169207 -> ciflow/trunk/169207 2025-12-04T08:57:44.6457350Z * [new tag] ciflow/trunk/169211 -> ciflow/trunk/169211 2025-12-04T08:57:44.6458221Z * [new tag] ciflow/trunk/169229 -> ciflow/trunk/169229 2025-12-04T08:57:44.6459246Z * [new tag] ciflow/trunk/169231 -> ciflow/trunk/169231 2025-12-04T08:57:44.6459992Z * [new tag] ciflow/trunk/169260 -> ciflow/trunk/169260 2025-12-04T08:57:44.6461007Z * [new tag] ciflow/trunk/169271 -> ciflow/trunk/169271 2025-12-04T08:57:44.6461835Z * [new tag] ciflow/trunk/169280 -> ciflow/trunk/169280 2025-12-04T08:57:44.6462514Z * [new tag] ciflow/trunk/169281 -> ciflow/trunk/169281 2025-12-04T08:57:44.6463253Z * [new tag] ciflow/trunk/169286 -> ciflow/trunk/169286 2025-12-04T08:57:44.6464206Z * [new tag] ciflow/trunk/169293 -> ciflow/trunk/169293 2025-12-04T08:57:44.6464941Z * [new tag] ciflow/trunk/169296 -> ciflow/trunk/169296 2025-12-04T08:57:44.6465689Z * [new tag] ciflow/trunk/169304 -> ciflow/trunk/169304 2025-12-04T08:57:44.6466473Z * [new tag] ciflow/trunk/169305 -> ciflow/trunk/169305 2025-12-04T08:57:44.6467206Z * [new tag] ciflow/trunk/169312 -> ciflow/trunk/169312 2025-12-04T08:57:44.6468784Z * [new tag] ciflow/trunk/169328 -> ciflow/trunk/169328 2025-12-04T08:57:44.6469489Z * [new tag] ciflow/trunk/169343 -> ciflow/trunk/169343 2025-12-04T08:57:44.6470255Z * [new tag] ciflow/trunk/169355 -> ciflow/trunk/169355 2025-12-04T08:57:44.6471042Z * [new tag] ciflow/trunk/169370 -> ciflow/trunk/169370 2025-12-04T08:57:44.6471956Z * [new tag] ciflow/trunk/169379 -> ciflow/trunk/169379 2025-12-04T08:57:44.6472749Z * [new tag] ciflow/trunk/169380 -> ciflow/trunk/169380 2025-12-04T08:57:44.6473499Z * [new tag] ciflow/trunk/169385 -> ciflow/trunk/169385 2025-12-04T08:57:44.6474381Z * [new tag] ciflow/trunk/169387 -> ciflow/trunk/169387 2025-12-04T08:57:44.6475353Z * [new tag] ciflow/trunk/169410 -> ciflow/trunk/169410 2025-12-04T08:57:44.6476098Z * [new tag] ciflow/trunk/169412 -> ciflow/trunk/169412 2025-12-04T08:57:44.6476879Z * [new tag] ciflow/trunk/169418 -> ciflow/trunk/169418 2025-12-04T08:57:44.6477648Z * [new tag] ciflow/trunk/169423 -> ciflow/trunk/169423 2025-12-04T08:57:44.6478433Z * [new tag] ciflow/trunk/169427 -> ciflow/trunk/169427 2025-12-04T08:57:44.6479232Z * [new tag] ciflow/trunk/169430 -> ciflow/trunk/169430 2025-12-04T08:57:44.6480095Z * [new tag] ciflow/trunk/169437 -> ciflow/trunk/169437 2025-12-04T08:57:44.6480865Z * [new tag] ciflow/trunk/169442 -> ciflow/trunk/169442 2025-12-04T08:57:44.6481609Z * [new tag] ciflow/trunk/169452 -> ciflow/trunk/169452 2025-12-04T08:57:44.6482346Z * [new tag] ciflow/trunk/169454 -> ciflow/trunk/169454 2025-12-04T08:57:44.6483086Z * [new tag] ciflow/trunk/169459 -> ciflow/trunk/169459 2025-12-04T08:57:44.6483983Z * [new tag] ciflow/trunk/169474 -> ciflow/trunk/169474 2025-12-04T08:57:44.6484737Z * [new tag] ciflow/trunk/169475 -> ciflow/trunk/169475 2025-12-04T08:57:44.6485463Z * [new tag] ciflow/trunk/169476 -> ciflow/trunk/169476 2025-12-04T08:57:44.6486299Z * [new tag] ciflow/trunk/169487 -> ciflow/trunk/169487 2025-12-04T08:57:44.6487067Z * [new tag] ciflow/trunk/169497 -> ciflow/trunk/169497 2025-12-04T08:57:44.6487881Z * [new tag] ciflow/trunk/169503 -> ciflow/trunk/169503 2025-12-04T08:57:44.6488610Z * [new tag] ciflow/trunk/169505 -> ciflow/trunk/169505 2025-12-04T08:57:44.6489361Z * [new tag] ciflow/trunk/169507 -> ciflow/trunk/169507 2025-12-04T08:57:44.6490097Z * [new tag] ciflow/trunk/169514 -> ciflow/trunk/169514 2025-12-04T08:57:44.6490853Z * [new tag] ciflow/trunk/169517 -> ciflow/trunk/169517 2025-12-04T08:57:44.6491660Z * [new tag] ciflow/trunk/169519 -> ciflow/trunk/169519 2025-12-04T08:57:44.6492315Z * [new tag] ciflow/trunk/169528 -> ciflow/trunk/169528 2025-12-04T08:57:44.6493050Z * [new tag] ciflow/trunk/169541 -> ciflow/trunk/169541 2025-12-04T08:57:44.6493926Z * [new tag] ciflow/trunk/169555 -> ciflow/trunk/169555 2025-12-04T08:57:44.6495544Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-12-04T08:57:44.6496330Z * [new tag] ciflow/vllm/165270 -> ciflow/vllm/165270 2025-12-04T08:57:44.6497047Z * [new tag] ciflow/vllm/165274 -> ciflow/vllm/165274 2025-12-04T08:57:44.6497767Z * [new tag] ciflow/vllm/166494 -> ciflow/vllm/166494 2025-12-04T08:57:44.6498597Z * [new tag] ciflow/vllm/169219 -> ciflow/vllm/169219 2025-12-04T08:57:44.6499264Z * [new tag] ciflow/vllm/169220 -> ciflow/vllm/169220 2025-12-04T08:57:44.6500124Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-12-04T08:57:44.6500833Z * [new tag] ciflow/xpu/159718 -> ciflow/xpu/159718 2025-12-04T08:57:44.6501548Z * [new tag] ciflow/xpu/161940 -> ciflow/xpu/161940 2025-12-04T08:57:44.6502364Z * [new tag] ciflow/xpu/163251 -> ciflow/xpu/163251 2025-12-04T08:57:44.6503074Z * [new tag] ciflow/xpu/166829 -> ciflow/xpu/166829 2025-12-04T08:57:44.6503792Z * [new tag] ciflow/xpu/166843 -> ciflow/xpu/166843 2025-12-04T08:57:44.6504539Z * [new tag] ciflow/xpu/167972 -> ciflow/xpu/167972 2025-12-04T08:57:44.6505243Z * [new tag] ciflow/xpu/167981 -> ciflow/xpu/167981 2025-12-04T08:57:44.6505962Z * [new tag] ciflow/xpu/168213 -> ciflow/xpu/168213 2025-12-04T08:57:44.6506647Z * [new tag] ciflow/xpu/168262 -> ciflow/xpu/168262 2025-12-04T08:57:44.6507437Z * [new tag] ciflow/xpu/168328 -> ciflow/xpu/168328 2025-12-04T08:57:44.6508683Z * [new tag] ciflow/xpu/168950 -> ciflow/xpu/168950 2025-12-04T08:57:44.6509930Z * [new tag] ciflow/xpu/169039 -> ciflow/xpu/169039 2025-12-04T08:57:44.6510774Z * [new tag] ciflow/xpu/169200 -> ciflow/xpu/169200 2025-12-04T08:57:44.6511557Z * [new tag] ciflow/xpu/169203 -> ciflow/xpu/169203 2025-12-04T08:57:44.6512311Z * [new tag] ciflow/xpu/169229 -> ciflow/xpu/169229 2025-12-04T08:57:44.6513098Z * [new tag] ciflow/xpu/169230 -> ciflow/xpu/169230 2025-12-04T08:57:44.6513938Z * [new tag] ciflow/xpu/169231 -> ciflow/xpu/169231 2025-12-04T08:57:44.6514831Z * [new tag] ciflow/xpu/169241 -> ciflow/xpu/169241 2025-12-04T08:57:44.6515597Z * [new tag] ciflow/xpu/169280 -> ciflow/xpu/169280 2025-12-04T08:57:44.6516386Z * [new tag] ciflow/xpu/169296 -> ciflow/xpu/169296 2025-12-04T08:57:44.6517261Z * [new tag] ciflow/xpu/169353 -> ciflow/xpu/169353 2025-12-04T08:57:44.6518034Z * [new tag] ciflow/xpu/169410 -> ciflow/xpu/169410 2025-12-04T08:57:44.6518794Z * [new tag] ciflow/xpu/169442 -> ciflow/xpu/169442 2025-12-04T08:57:44.6519612Z * [new tag] ciflow/xpu/169555 -> ciflow/xpu/169555 2025-12-04T08:57:44.6520534Z * [new tag] cslpull75 -> cslpull75 2025-12-04T08:57:44.6521393Z * [new tag] cslpull76 -> cslpull76 2025-12-04T08:57:44.6522216Z * [new tag] cslpull77 -> cslpull77 2025-12-04T08:57:44.6523039Z * [new tag] cslpull78 -> cslpull78 2025-12-04T08:57:44.6525580Z * [new tag] cslpull79 -> cslpull79 2025-12-04T08:57:44.6526622Z * [new tag] cslpull80 -> cslpull80 2025-12-04T08:57:44.6527586Z * [new tag] cslpull81 -> cslpull81 2025-12-04T08:57:44.6528568Z * [new tag] cslpull82 -> cslpull82 2025-12-04T08:57:44.6529403Z * [new tag] cslpull83 -> cslpull83 2025-12-04T08:57:44.6530256Z * [new tag] cslpull84 -> cslpull84 2025-12-04T08:57:44.6531082Z * [new tag] cslpull85 -> cslpull85 2025-12-04T08:57:44.6532005Z * [new tag] cslpull86 -> cslpull86 2025-12-04T08:57:44.6532871Z * [new tag] cslpull87 -> cslpull87 2025-12-04T08:57:44.6533804Z * [new tag] cslpull88 -> cslpull88 2025-12-04T08:57:44.6534660Z * [new tag] cslpull89 -> cslpull89 2025-12-04T08:57:44.6535364Z * [new tag] cslpull90 -> cslpull90 2025-12-04T08:57:44.6536845Z * [new tag] cslpull91 -> cslpull91 2025-12-04T08:57:44.6537594Z * [new tag] cslpull92 -> cslpull92 2025-12-04T08:57:44.6538491Z * [new tag] flight_5 -> flight_5 2025-12-04T08:57:44.6539536Z * [new tag] flight_5.1 -> flight_5.1 2025-12-04T08:57:44.6540354Z * [new tag] flight_5.2 -> flight_5.2 2025-12-04T08:57:44.6541268Z * [new tag] flight_5.3 -> flight_5.3 2025-12-04T08:57:44.6542160Z * [new tag] forpull1 -> forpull1 2025-12-04T08:57:44.6543214Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-12-04T08:57:44.6544047Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-12-04T08:57:44.6544900Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-12-04T08:57:44.6545814Z * [new tag] nightly-binary -> nightly-binary 2025-12-04T08:57:44.6546827Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-12-04T08:57:44.6548127Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-12-04T08:57:44.6549439Z * [new tag] trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 -> trunk/02d8bd6974cf84b721680d773dbdb1b6f40ce272 2025-12-04T08:57:44.6550476Z * [new tag] trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e -> trunk/066997fb38ade71e00d78e9d572e380b5f02bd3e 2025-12-04T08:57:44.6551554Z * [new tag] trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 -> trunk/076e7b19fa1d481ad778d06d2b49ba57d3ce8c88 2025-12-04T08:57:44.6552666Z * [new tag] trunk/07dcc0b83db3211653a38565a24e15acdba75654 -> trunk/07dcc0b83db3211653a38565a24e15acdba75654 2025-12-04T08:57:44.6553713Z * [new tag] trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb -> trunk/082e96b68dfcd16cab7cfafc4d3d055767dab3eb 2025-12-04T08:57:44.6554745Z * [new tag] trunk/088048f2fea28ff7d450f65c72419ca45780d30b -> trunk/088048f2fea28ff7d450f65c72419ca45780d30b 2025-12-04T08:57:44.6555760Z * [new tag] trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 -> trunk/09076941a95c76f4d9ad189d064dfd8baa39e672 2025-12-04T08:57:44.6556765Z * [new tag] trunk/0b80a4c62b94402844bf221791c096b0035c6d75 -> trunk/0b80a4c62b94402844bf221791c096b0035c6d75 2025-12-04T08:57:44.6558004Z * [new tag] trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 -> trunk/0bbbdf1750567a980634ad907a325357ba8ba8f2 2025-12-04T08:57:44.6559084Z * [new tag] trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 -> trunk/0c281dd78773b2bc17c58ead0e4cd4ac46e775c5 2025-12-04T08:57:44.6560324Z * [new tag] trunk/135f3753c418a6879b1954904184937b67e61688 -> trunk/135f3753c418a6879b1954904184937b67e61688 2025-12-04T08:57:44.6561388Z * [new tag] trunk/15da21026cb13cd20257dc9e96830db108743c10 -> trunk/15da21026cb13cd20257dc9e96830db108743c10 2025-12-04T08:57:44.6562378Z * [new tag] trunk/166efdad2ac827f30fb02504c6017520257f88ec -> trunk/166efdad2ac827f30fb02504c6017520257f88ec 2025-12-04T08:57:44.6563345Z * [new tag] trunk/174272c15fae553d8488140af931f7d8050a313f -> trunk/174272c15fae553d8488140af931f7d8050a313f 2025-12-04T08:57:44.6564336Z * [new tag] trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 -> trunk/18f3ca08f13b8de61307f5e8cd7d4cccb67e9d11 2025-12-04T08:57:44.6565349Z * [new tag] trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 -> trunk/1902eddfe655a15ebcf2c72bd81ade110fdeef63 2025-12-04T08:57:44.6566350Z * [new tag] trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 -> trunk/195f92e98d3d66738577f11f22c4b5c8a1c76dd5 2025-12-04T08:57:44.6567347Z * [new tag] trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 -> trunk/1aa13e17de39e3c768ea7aebaad166ce72a06676 2025-12-04T08:57:44.6568395Z * [new tag] trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e -> trunk/1afe2832f58e24e54a5bfda5a5afa9b96fdea40e 2025-12-04T08:57:44.6569394Z * [new tag] trunk/1c87554d74140eaee964ca8b1832cede67f5f520 -> trunk/1c87554d74140eaee964ca8b1832cede67f5f520 2025-12-04T08:57:44.6570389Z * [new tag] trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 -> trunk/1ccb743b7b5be955f49736c162c4f5004b8a0dd8 2025-12-04T08:57:44.6571371Z * [new tag] trunk/1cee47d6ce0a02227185b566593f002dd639ca0c -> trunk/1cee47d6ce0a02227185b566593f002dd639ca0c 2025-12-04T08:57:44.6572366Z * [new tag] trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d -> trunk/1d21b4df2babe322e5d085ceb6de884eb260a62d 2025-12-04T08:57:44.6573372Z * [new tag] trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 -> trunk/1e34fb2550e4aa650314f7a6d9f6daf4da7478a8 2025-12-04T08:57:44.6574373Z * [new tag] trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de -> trunk/1e526fb5b1d93bfc70691c5c3955fdffc1b7b7de 2025-12-04T08:57:44.6575383Z * [new tag] trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 -> trunk/1ee32a8b1f554a312d79bad01ded24f38cd95543 2025-12-04T08:57:44.6576372Z * [new tag] trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 -> trunk/201e2c4117eb9744594dad6a5c18213d7b4705d7 2025-12-04T08:57:44.6577357Z * [new tag] trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f -> trunk/2353a0f60eb4b4cb6675907a7fa9fbedc1c02e7f 2025-12-04T08:57:44.6578339Z * [new tag] trunk/285779b1621cf9f073a062b0889a642d200308d9 -> trunk/285779b1621cf9f073a062b0889a642d200308d9 2025-12-04T08:57:44.6579312Z * [new tag] trunk/2887faaec6295d081580d09fce161201826c6d87 -> trunk/2887faaec6295d081580d09fce161201826c6d87 2025-12-04T08:57:44.6580296Z * [new tag] trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc -> trunk/296e67c92635443c67b11c0ae1bd045f03ebb7bc 2025-12-04T08:57:44.6581265Z * [new tag] trunk/29856679769b3dede478767e2fe6cfb51197cb25 -> trunk/29856679769b3dede478767e2fe6cfb51197cb25 2025-12-04T08:57:44.6582249Z * [new tag] trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 -> trunk/29e5455a4740c326ab187c7aa7b5ef98034ea563 2025-12-04T08:57:44.6583238Z * [new tag] trunk/2ac3ef882afb23136adc188975f0a8802fc68adf -> trunk/2ac3ef882afb23136adc188975f0a8802fc68adf 2025-12-04T08:57:44.6584224Z * [new tag] trunk/2bec68e73b64715354af076ad309335f943e36cd -> trunk/2bec68e73b64715354af076ad309335f943e36cd 2025-12-04T08:57:44.6585192Z * [new tag] trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 -> trunk/2c87367e6f88662cd5cedbd1537748b7948c38e1 2025-12-04T08:57:44.6586235Z * [new tag] trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 -> trunk/2d1f78fe3ec13820f136a2e0336da12a25f41708 2025-12-04T08:57:44.6587349Z * [new tag] trunk/2df6058f116a65722a0e03073402feb242572d35 -> trunk/2df6058f116a65722a0e03073402feb242572d35 2025-12-04T08:57:44.6588524Z * [new tag] trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec -> trunk/2e0c2e170fe658c440775c8e5c44228aafcc47ec 2025-12-04T08:57:44.6589550Z * [new tag] trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 -> trunk/2f9b7dad7b5419b063bd0f2e204de192720ebb94 2025-12-04T08:57:44.6590552Z * [new tag] trunk/305168768a95d69c444df5cd334bb774edfe06f1 -> trunk/305168768a95d69c444df5cd334bb774edfe06f1 2025-12-04T08:57:44.6591563Z * [new tag] trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 -> trunk/31fc12773026e8e00f054dd79ad9b2491e693b48 2025-12-04T08:57:44.6592585Z * [new tag] trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 -> trunk/320de0c6b0a3e7c6d2693ea5c28d5d0156ba7991 2025-12-04T08:57:44.6593619Z * [new tag] trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 -> trunk/3418bd29475dff06695045fcdf93e7d0dac67da8 2025-12-04T08:57:44.6594636Z * [new tag] trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf -> trunk/34a98608afa0cb5b48f0d6d30432fdd0a2614ddf 2025-12-04T08:57:44.6595678Z * [new tag] trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee -> trunk/35b7a9a26c5923d98aebaa41a031dae21788a9ee 2025-12-04T08:57:44.6596706Z * [new tag] trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 -> trunk/39d07dbf03a911bdd45d1af78d8638dc92074938 2025-12-04T08:57:44.6597725Z * [new tag] trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 -> trunk/3cd98b4205ada151042cc7ff097a82d4a4b18725 2025-12-04T08:57:44.6598757Z * [new tag] trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae -> trunk/3d35fd20a78ff4d016fa80f4e5fad37191d7bcae 2025-12-04T08:57:44.6599889Z * [new tag] trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f -> trunk/409a5fee945c46a3edaf5df162812f201bfd7b2f 2025-12-04T08:57:44.6600895Z * [new tag] trunk/42e9005cda22da3f1c559c3649218cebd671027c -> trunk/42e9005cda22da3f1c559c3649218cebd671027c 2025-12-04T08:57:44.6601895Z * [new tag] trunk/43b94713bbf340d3c124fde02d0f73add4021247 -> trunk/43b94713bbf340d3c124fde02d0f73add4021247 2025-12-04T08:57:44.6602888Z * [new tag] trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c -> trunk/44ac69388a4a5eb463dbd2a13f00d1e3b924566c 2025-12-04T08:57:44.6603873Z * [new tag] trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a -> trunk/45d14e2497292be06ad36eaa1aaaf7c630a2586a 2025-12-04T08:57:44.6604861Z * [new tag] trunk/45d310ad84854dff730c0b12e577d7998d978686 -> trunk/45d310ad84854dff730c0b12e577d7998d978686 2025-12-04T08:57:44.6605851Z * [new tag] trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 -> trunk/47b28ddf7bd74b50fa93b307a7d3b183a6d77f54 2025-12-04T08:57:44.6606853Z * [new tag] trunk/481e5ab336275bd3acd5fa8a611b05b4469012af -> trunk/481e5ab336275bd3acd5fa8a611b05b4469012af 2025-12-04T08:57:44.6607845Z * [new tag] trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 -> trunk/491731647f6b8a9345dcfb3bc9416aea254a7d96 2025-12-04T08:57:44.6608824Z * [new tag] trunk/49a04d26088acc17d948ddd66920f3e16371e873 -> trunk/49a04d26088acc17d948ddd66920f3e16371e873 2025-12-04T08:57:44.6609809Z * [new tag] trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 -> trunk/4bebc827c47d2f1f0fa1a417a5201a97aef3d985 2025-12-04T08:57:44.6610804Z * [new tag] trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f -> trunk/4c246677784c6a14bc2dbb9ff8773ef0a3a3222f 2025-12-04T08:57:44.6611802Z * [new tag] trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa -> trunk/4cfb47ff548b6d996641058cf04a70e311a4c3aa 2025-12-04T08:57:44.6612789Z * [new tag] trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c -> trunk/4e0061c1aa52f606dda8cfab0bd7591e588faf2c 2025-12-04T08:57:44.6613859Z * [new tag] trunk/4fefb8e7e942386ffac764a41b232241f82bea3a -> trunk/4fefb8e7e942386ffac764a41b232241f82bea3a 2025-12-04T08:57:44.6614894Z * [new tag] trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d -> trunk/503b2640023521f5a35cd9a52fc8033d73a95d0d 2025-12-04T08:57:44.6615891Z * [new tag] trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 -> trunk/518c2b1b3dab9a2ef2849e04b3bc2f20c1c41db9 2025-12-04T08:57:44.6616876Z * [new tag] trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 -> trunk/5191b2fa68ba19960912bfd7fd721c79d76bb1f3 2025-12-04T08:57:44.6617885Z * [new tag] trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a -> trunk/52ac0f0dc4acacd219f1317fbc28ec631c01e07a 2025-12-04T08:57:44.6618882Z * [new tag] trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 -> trunk/539ba711b029de9f191070f4f0d12f18f5b7f292 2025-12-04T08:57:44.6619884Z * [new tag] trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 -> trunk/556375b55deebebbc56cb7aef81f4d52f031ba28 2025-12-04T08:57:44.6620871Z * [new tag] trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 -> trunk/55c4ab554845481d0a69a3811937575fe8bb1a66 2025-12-04T08:57:44.6621839Z * [new tag] trunk/5634469fda9e5d98869c82c7d03bb08914245f96 -> trunk/5634469fda9e5d98869c82c7d03bb08914245f96 2025-12-04T08:57:44.6622808Z * [new tag] trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc -> trunk/5778f6ff894686a975a9a23645178ae4c87ad5dc 2025-12-04T08:57:44.6624124Z * [new tag] trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 -> trunk/587d63a3e07de5dc91065f9ef70bcacda9989068 2025-12-04T08:57:44.6625146Z * [new tag] trunk/597930f6b568852356ca9795dac76f9e4653adbd -> trunk/597930f6b568852356ca9795dac76f9e4653adbd 2025-12-04T08:57:44.6626156Z * [new tag] trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 -> trunk/597df3a4e2a67b9fdbe1a89b2f4d74f822274db6 2025-12-04T08:57:44.6627192Z * [new tag] trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 -> trunk/59abd50e931f4efb21b053f7a2911f5d8a49d883 2025-12-04T08:57:44.6628304Z * [new tag] trunk/5a607febc04c3a2b5824c75f3f60307867439a2c -> trunk/5a607febc04c3a2b5824c75f3f60307867439a2c 2025-12-04T08:57:44.6629316Z * [new tag] trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b -> trunk/5bf1cdf4755c54ef462b44cb8041b0a57311556b 2025-12-04T08:57:44.6630325Z * [new tag] trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c -> trunk/5f0030ba63d334d7e8c93a09e41403b89e4c573c 2025-12-04T08:57:44.6631336Z * [new tag] trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 -> trunk/5f21d27e71268464d362a96c9ac09ea475f7f202 2025-12-04T08:57:44.6632347Z * [new tag] trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 -> trunk/5fafc13038c9988d9ac21fa793fbd5890604b447 2025-12-04T08:57:44.6633378Z * [new tag] trunk/61be54a31dc09b59d99b62176fb935aee0b924ef -> trunk/61be54a31dc09b59d99b62176fb935aee0b924ef 2025-12-04T08:57:44.6634376Z * [new tag] trunk/62d3ccd71484ed6a760d909b41487101bbc65719 -> trunk/62d3ccd71484ed6a760d909b41487101bbc65719 2025-12-04T08:57:44.6635392Z * [new tag] trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b -> trunk/641cdb68ae27668eb441d0e49c87a0602c120c2b 2025-12-04T08:57:44.6636402Z * [new tag] trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a -> trunk/65c4620d6bb0c6029f69762c22b91dda2294da9a 2025-12-04T08:57:44.6637406Z * [new tag] trunk/66004b993744b4106bf8afaba71f3c228a804206 -> trunk/66004b993744b4106bf8afaba71f3c228a804206 2025-12-04T08:57:44.6638422Z * [new tag] trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 -> trunk/6658a04c7ca67acb64512341342e7b3ee13ee386 2025-12-04T08:57:44.6639529Z * [new tag] trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 -> trunk/6864e309092a71f8ab0ca6a4dc7f8a4073fd31c4 2025-12-04T08:57:44.6640683Z * [new tag] trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d -> trunk/6c261c6cb07892c90ca19ed51c9705b1659a3f7d 2025-12-04T08:57:44.6641731Z * [new tag] trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b -> trunk/6c8b6a043f1628188b6396b3a2a6e000ca68362b 2025-12-04T08:57:44.6642712Z * [new tag] trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 -> trunk/6ceb4a32f92ae67ce5d7d97931d17401ebf5ffa5 2025-12-04T08:57:44.6643696Z * [new tag] trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 -> trunk/6e404e9b7d6f5fb0de86aa73888c3038248c17f8 2025-12-04T08:57:44.6644691Z * [new tag] trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec -> trunk/6ec30b490aee1db6bcdc7340abddef25784f08ec 2025-12-04T08:57:44.6645694Z * [new tag] trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 -> trunk/6f2783a6c08e1db34275ff25176ffe9aebc30a71 2025-12-04T08:57:44.6646695Z * [new tag] trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d -> trunk/6f53fefeb90ad3281119b5cfc4aa9ffd8a066e3d 2025-12-04T08:57:44.6647693Z * [new tag] trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a -> trunk/6f7dcf51e46d0c880db1a2f5c70de57adb576f4a 2025-12-04T08:57:44.6648698Z * [new tag] trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e -> trunk/6ff831180d2fa436c7f1c1af3adac641fce9d60e 2025-12-04T08:57:44.6649690Z * [new tag] trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 -> trunk/70076464a63ab218a7ceefb0e76ccd7131deb8f8 2025-12-04T08:57:44.6650671Z * [new tag] trunk/70d797a5fc109b20a517646fcaa819477cd0d485 -> trunk/70d797a5fc109b20a517646fcaa819477cd0d485 2025-12-04T08:57:44.6651636Z * [new tag] trunk/7348cb355ff0a6f79cd4871215aea72185748734 -> trunk/7348cb355ff0a6f79cd4871215aea72185748734 2025-12-04T08:57:44.6652612Z * [new tag] trunk/74fe26a1ebe32931783569f2e762e3c2c974901f -> trunk/74fe26a1ebe32931783569f2e762e3c2c974901f 2025-12-04T08:57:44.6653615Z * [new tag] trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 -> trunk/76aeb8c7e0f795b3fddca134cbea9a69da3ee696 2025-12-04T08:57:44.6654616Z * [new tag] trunk/7741edd4ed665f3988052e260863efb508d61a03 -> trunk/7741edd4ed665f3988052e260863efb508d61a03 2025-12-04T08:57:44.6655595Z * [new tag] trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 -> trunk/78adb3b3df41b45d2368b67226d2f864b78939a6 2025-12-04T08:57:44.6656581Z * [new tag] trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 -> trunk/79d7b178225e5ed24d4e1db74e5abbff848f5fb7 2025-12-04T08:57:44.6657564Z * [new tag] trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 -> trunk/7a1e316115fc6996b3f2336822ba5d5f6179f0c3 2025-12-04T08:57:44.6658545Z * [new tag] trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca -> trunk/7a41b66367c38d0af3e8a90f7be48d6b281e7bca 2025-12-04T08:57:44.6659527Z * [new tag] trunk/7b7af390ea8541c611d1ce2018a6934188fc197b -> trunk/7b7af390ea8541c611d1ce2018a6934188fc197b 2025-12-04T08:57:44.6660496Z * [new tag] trunk/7ba4680f3755a560af81aa0f688791e367aa3609 -> trunk/7ba4680f3755a560af81aa0f688791e367aa3609 2025-12-04T08:57:44.6661491Z * [new tag] trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b -> trunk/7bc2a66ded06a0b2549aa51d807edc5dc3e73d1b 2025-12-04T08:57:44.6662484Z * [new tag] trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 -> trunk/7c648509a7470ace9fb2bae960dd4790f7e943e9 2025-12-04T08:57:44.6663487Z * [new tag] trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 -> trunk/7cbc2d034cecd21ab5c9707d0a9c525c17143fb8 2025-12-04T08:57:44.6664490Z * [new tag] trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed -> trunk/7d1bbaf4ba301ea3fba6f3c7bc02d58f6417aaed 2025-12-04T08:57:44.6665496Z * [new tag] trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 -> trunk/7d2a33e4ebf60b217a3cd77feae19231eb996fc8 2025-12-04T08:57:44.6666563Z * [new tag] trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e -> trunk/7eb625920054b1126a7d2d99818aaa188c6ba95e 2025-12-04T08:57:44.6667791Z * [new tag] trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead -> trunk/7f55ba19c456a3d6cc443dd9edb6bb7cca677ead 2025-12-04T08:57:44.6668876Z * [new tag] trunk/81af382128efa094d8702e18f2c133760904c718 -> trunk/81af382128efa094d8702e18f2c133760904c718 2025-12-04T08:57:44.6669881Z * [new tag] trunk/84149583d483e9c973c9a0feda70e4f3964947b0 -> trunk/84149583d483e9c973c9a0feda70e4f3964947b0 2025-12-04T08:57:44.6670882Z * [new tag] trunk/85a315917efe82c24306be805c584ec044951c75 -> trunk/85a315917efe82c24306be805c584ec044951c75 2025-12-04T08:57:44.6671889Z * [new tag] trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece -> trunk/87329491c82a5f8c1cc4ec11d8f55a5de2551ece 2025-12-04T08:57:44.6672888Z * [new tag] trunk/892640e25aeefa8007c5af837214b4502b6b62a6 -> trunk/892640e25aeefa8007c5af837214b4502b6b62a6 2025-12-04T08:57:44.6673918Z * [new tag] trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 -> trunk/89e3bbcb5b5321dc8b9520b4d5a8ee60cea1d0b4 2025-12-04T08:57:44.6674945Z * [new tag] trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c -> trunk/8c73bbbb02159223c0c97d268a0a74cb78158a1c 2025-12-04T08:57:44.6675979Z * [new tag] trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 -> trunk/8d56e98c8db988a22cb2dfaeefb30bc7d2a3cc43 2025-12-04T08:57:44.6677008Z * [new tag] trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 -> trunk/8d9dd9603e5ee26c01007f0cd4f018e584840922 2025-12-04T08:57:44.6678007Z * [new tag] trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca -> trunk/8ef0c0b02b062d75e7c9be2594914a3e784d23ca 2025-12-04T08:57:44.6679028Z * [new tag] trunk/90b27e7e8352cde97d32ddad24740ef819633f38 -> trunk/90b27e7e8352cde97d32ddad24740ef819633f38 2025-12-04T08:57:44.6680131Z * [new tag] trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 -> trunk/90f0139e64b2951815d524b6a373bed20c4fbf90 2025-12-04T08:57:44.6681111Z * [new tag] trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c -> trunk/93d0d6838c56af59b0dba794e6aa08f0c1c7799c 2025-12-04T08:57:44.6682104Z * [new tag] trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 -> trunk/94ca8d5f1e81fea3ae488650a0fb6795049a9f87 2025-12-04T08:57:44.6683102Z * [new tag] trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 -> trunk/9844fbeadd5cebdf1281d6fbf79164139c352693 2025-12-04T08:57:44.6684087Z * [new tag] trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa -> trunk/99024dec888ec1e50b546822a32b6fb2f35e5eaa 2025-12-04T08:57:44.6685076Z * [new tag] trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d -> trunk/9a296e640fc88aa44d275b48cd9cc30c573b169d 2025-12-04T08:57:44.6686067Z * [new tag] trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 -> trunk/9b3e34d8589b29f7b4e7fab6f78711b7ca6e4639 2025-12-04T08:57:44.6687041Z * [new tag] trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 -> trunk/9cd055e547e9b67a5f9827f8999c38d7eda1bcb8 2025-12-04T08:57:44.6688029Z * [new tag] trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d -> trunk/9f0df5686cb4ada94f94620acba2e3c3f363b11d 2025-12-04T08:57:44.6689025Z * [new tag] trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a -> trunk/9f7fceb887d0cfa0326a59b887821c63ff11340a 2025-12-04T08:57:44.6690007Z * [new tag] trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 -> trunk/9f8ef8855d3078d70f7b782540ff2aaf158d6742 2025-12-04T08:57:44.6690981Z * [new tag] trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 -> trunk/9fb52efc797b47a1f425a03aa5e47b866d8b1098 2025-12-04T08:57:44.6691975Z * [new tag] trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa -> trunk/9ff4a2ebc5762d46c73e46b1b523d7ff349fedfa 2025-12-04T08:57:44.6692956Z * [new tag] trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d -> trunk/a0f3937b94422354538ebbd47202d5b0e8a3fd0d 2025-12-04T08:57:44.6694154Z * [new tag] trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c -> trunk/a15066c28b3145e6edbfc88359d0411d14cfc70c 2025-12-04T08:57:44.6695180Z * [new tag] trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 -> trunk/a20f775e82564d2a9979221ed7f3b8d7cf54ce90 2025-12-04T08:57:44.6696185Z * [new tag] trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c -> trunk/a2973fb00ec002dd4b6bbf07385f066efb259b8c 2025-12-04T08:57:44.6697181Z * [new tag] trunk/a7dc6dab9ad911259d4801c502907e531594db45 -> trunk/a7dc6dab9ad911259d4801c502907e531594db45 2025-12-04T08:57:44.6698183Z * [new tag] trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 -> trunk/a951a9cee65c01660bbc6e6fded90ecb10fa6109 2025-12-04T08:57:44.6699177Z * [new tag] trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e -> trunk/abfa1a6d65c7c159e35c72c25979b9da4971689e 2025-12-04T08:57:44.6700170Z * [new tag] trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e -> trunk/ae3a2395bf66151078e2d201716f7d63ce1c6f3e 2025-12-04T08:57:44.6701162Z * [new tag] trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e -> trunk/afdff7f0325080dedac44d080cb5a3b0e65e6c5e 2025-12-04T08:57:44.6702176Z * [new tag] trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 -> trunk/b1aed4e7a72c03a38f44543aaea0dae2e9b76d48 2025-12-04T08:57:44.6703173Z * [new tag] trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 -> trunk/b1decff555cd50e2123c8c6e25cc0d447c411f62 2025-12-04T08:57:44.6704176Z * [new tag] trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 -> trunk/b2b6b034c9fd08672c40e63ef243556ad4c49bd2 2025-12-04T08:57:44.6705155Z * [new tag] trunk/b39813b4a04931682b0491adba2138d01d716d99 -> trunk/b39813b4a04931682b0491adba2138d01d716d99 2025-12-04T08:57:44.6706129Z * [new tag] trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 -> trunk/b3a7edb2311367974cc7cd764cfb11a5d6758b24 2025-12-04T08:57:44.6707130Z * [new tag] trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 -> trunk/b4cc1329c86acaef6d42c1fac7169b8d870ab0d7 2025-12-04T08:57:44.6708435Z * [new tag] trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a -> trunk/b555c39217f765759954a4f9f9bd1e9b87bed11a 2025-12-04T08:57:44.6708888Z * [new tag] trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 -> trunk/b6b6c80379388b7f9932c3e6a0f9907bf430e417 2025-12-04T08:57:44.6709333Z * [new tag] trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 -> trunk/b6b6d912df0b6f4082f8e50b18bd1de1dd7325f4 2025-12-04T08:57:44.6709778Z * [new tag] trunk/b7d60685f8cbc939b68a20871e90db67e729329b -> trunk/b7d60685f8cbc939b68a20871e90db67e729329b 2025-12-04T08:57:44.6710220Z * [new tag] trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e -> trunk/b7f6b9a4fc6259f7af068f31868b3119bb1bac3e 2025-12-04T08:57:44.6710659Z * [new tag] trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf -> trunk/b8c4ba3593761e7b2a3ebd86f040fb07b47c02cf 2025-12-04T08:57:44.6711116Z * [new tag] trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 -> trunk/b9c8f3a4884befb965ff42620ce44a71b04887f5 2025-12-04T08:57:44.6711551Z * [new tag] trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f -> trunk/ba1412546f3082c0958c077acc2025e4dbc33f1f 2025-12-04T08:57:44.6712012Z * [new tag] trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f -> trunk/bac403c0b38c63bdbcc0c31f1c2b0bc0260f610f 2025-12-04T08:57:44.6712448Z * [new tag] trunk/bb3034198b459401fabeab254e1b99f0115046e2 -> trunk/bb3034198b459401fabeab254e1b99f0115046e2 2025-12-04T08:57:44.6712884Z * [new tag] trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 -> trunk/bc39b2b3bc7a6e19a42e62bd576974035086fe55 2025-12-04T08:57:44.6713335Z * [new tag] trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 -> trunk/bc43d5b297f207a11d83d77ddf0152bdaabe15a8 2025-12-04T08:57:44.6713838Z * [new tag] trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 -> trunk/bc6a4863c7246a6493d16d4ea6eee71ec07c6a09 2025-12-04T08:57:44.6714359Z * [new tag] trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 -> trunk/bea4912944defdbcb8b061800caab6cbbbd01df5 2025-12-04T08:57:44.6714794Z * [new tag] trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 -> trunk/c04e2c656f48d82d1521b867bbbf03967b9b7564 2025-12-04T08:57:44.6715219Z * [new tag] trunk/c0660bcee27e7d7731634e274576a7081882bede -> trunk/c0660bcee27e7d7731634e274576a7081882bede 2025-12-04T08:57:44.6715681Z * [new tag] trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac -> trunk/c178ed43d3d99cbefe84fbfb21d6f282b20d62ac 2025-12-04T08:57:44.6716115Z * [new tag] trunk/c55b1e8f61d041ee436d697449eb028931d574fb -> trunk/c55b1e8f61d041ee436d697449eb028931d574fb 2025-12-04T08:57:44.6716558Z * [new tag] trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 -> trunk/c6ae7579fe12fe75f1a8f7043a494c90567273f1 2025-12-04T08:57:44.6717010Z * [new tag] trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 -> trunk/c8210e7d94bad5ae21ac389fa4ba8a463c76c4d0 2025-12-04T08:57:44.6717446Z * [new tag] trunk/cc0853af42122f8185321f542616f4474e717f09 -> trunk/cc0853af42122f8185321f542616f4474e717f09 2025-12-04T08:57:44.6717894Z * [new tag] trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 -> trunk/cddec6562eabfa390d014fa3741a5659cf9c94c9 2025-12-04T08:57:44.6718386Z * [new tag] trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a -> trunk/ce5e7e3bf1f4b69a4f4f93d288ba75b906df492a 2025-12-04T08:57:44.6719383Z * [new tag] trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace -> trunk/d038b0130ec7c20ebcac219301292fd8e98a1ace 2025-12-04T08:57:44.6720323Z * [new tag] trunk/d16447dacaf2420ea175f0c275c75da951f57d39 -> trunk/d16447dacaf2420ea175f0c275c75da951f57d39 2025-12-04T08:57:44.6721297Z * [new tag] trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 -> trunk/d19f1e8cab6810bb2e99141f9976665954c67a50 2025-12-04T08:57:44.6722166Z * [new tag] trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 -> trunk/d1c9f03b2a5af4104721712f8cdffe9b4f340c01 2025-12-04T08:57:44.6723159Z * [new tag] trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf -> trunk/d40f4950f2b7f7aa380a22fe0f6166e71680fbcf 2025-12-04T08:57:44.6724404Z * [new tag] trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 -> trunk/d5038950bacfe36bbf24a47a455fe76901deb8e8 2025-12-04T08:57:44.6725353Z * [new tag] trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d -> trunk/d54ff42903c2ae0533931ff11d23b35f875bdb3d 2025-12-04T08:57:44.6726367Z * [new tag] trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 -> trunk/d76697633a2d2b9cced1ae21161849b33bfe7e47 2025-12-04T08:57:44.6727299Z * [new tag] trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 -> trunk/d78f52b199c547106d4cd9d2856dd0805c118bf1 2025-12-04T08:57:44.6728250Z * [new tag] trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e -> trunk/d8fd5c6eed28e5004150691d048a3f6785e19a8e 2025-12-04T08:57:44.6729335Z * [new tag] trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a -> trunk/d900f5e86745dec76713f4b0ef07005ef36b2f5a 2025-12-04T08:57:44.6730183Z * [new tag] trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b -> trunk/d973dc6b87d763859fe1c5bd1287e3b6b1c49d1b 2025-12-04T08:57:44.6731133Z * [new tag] trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec -> trunk/d998c03304cb6ede76e1ed535b4ddeb6c2bf40ec 2025-12-04T08:57:44.6732115Z * [new tag] trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf -> trunk/d9cb8a70833101dbbe16b99520cfbdd70d0a87bf 2025-12-04T08:57:44.6733043Z * [new tag] trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd -> trunk/d9d5e91b43f70eb8637af55db6856d49be391ffd 2025-12-04T08:57:44.6733986Z * [new tag] trunk/dd18a75336a4fbd7497955cc5665904724fce889 -> trunk/dd18a75336a4fbd7497955cc5665904724fce889 2025-12-04T08:57:44.6735049Z * [new tag] trunk/ded9bcd61a059bf723e6e84689552962b480ea77 -> trunk/ded9bcd61a059bf723e6e84689552962b480ea77 2025-12-04T08:57:44.6736386Z * [new tag] trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c -> trunk/dfbd3714d15c37a7b83b322a6b60f997fc00f50c 2025-12-04T08:57:44.6737545Z * [new tag] trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b -> trunk/e115f9f4e4b039f8e9a642aaa2bd8254a920541b 2025-12-04T08:57:44.6738283Z * [new tag] trunk/e3f24fd73ad74c6e7176687986436956c7c18235 -> trunk/e3f24fd73ad74c6e7176687986436956c7c18235 2025-12-04T08:57:44.6739306Z * [new tag] trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e -> trunk/e7d24d3ff93d1503ba63860b7057438ad93f918e 2025-12-04T08:57:44.6740353Z * [new tag] trunk/ea7035f462a0d2830865ee86c832bd101e1427fc -> trunk/ea7035f462a0d2830865ee86c832bd101e1427fc 2025-12-04T08:57:44.6741165Z * [new tag] trunk/eabb7ad2128580ef674446027b95bcf4e21e8df3 -> trunk/eabb7ad2128580ef674446027b95bcf4e21e8df3 2025-12-04T08:57:44.6742117Z * [new tag] trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf -> trunk/eb5c63652a33da42e7018c23df5f20a3eb4c6ccf 2025-12-04T08:57:44.6743055Z * [new tag] trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e -> trunk/ec2c71f5c85021b8938cdafadce24c15a36fd93e 2025-12-04T08:57:44.6743959Z * [new tag] trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e -> trunk/ecbcc3f6bf327856b435b259ac63cc2f328c4b4e 2025-12-04T08:57:44.6745423Z * [new tag] trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 -> trunk/ee87bbe876c42575e961b32a0827d76bc9782ca2 2025-12-04T08:57:44.6746272Z * [new tag] trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 -> trunk/ef019d1d431c4c5a95b594cb90d40a50cd00f5e4 2025-12-04T08:57:44.6747519Z * [new tag] trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 -> trunk/ef8ecc13830a86c4b231f1aad9aba7851db61b53 2025-12-04T08:57:44.6748732Z * [new tag] trunk/f1076f5510920044912247b1abb8760cb820f598 -> trunk/f1076f5510920044912247b1abb8760cb820f598 2025-12-04T08:57:44.6749666Z * [new tag] trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 -> trunk/f2d6a75a00a1d648ca9a0abc6a33e14c3dea6c40 2025-12-04T08:57:44.6750609Z * [new tag] trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 -> trunk/f47dd0ddef1359e5b43e4b962412f67b30ecde56 2025-12-04T08:57:44.6751674Z * [new tag] trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 -> trunk/f49d32dfa4730dcfb1b60eeeb369b5889da983c8 2025-12-04T08:57:44.6752553Z * [new tag] trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 -> trunk/f4dedf78fc30fd4b93975787ca6074ee89db9467 2025-12-04T08:57:44.6753474Z * [new tag] trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 -> trunk/f7c0d03819ebed05c4038f095d66d1b8c54aca17 2025-12-04T08:57:44.6754432Z * [new tag] trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 -> trunk/f7e1bd80a063e17453c361837ba6ea2570920a73 2025-12-04T08:57:44.6755242Z * [new tag] trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 -> trunk/f9bd6c53624c7c0ea3772de78498326e84c2f0e7 2025-12-04T08:57:44.6756298Z * [new tag] trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b -> trunk/fb5be221a46b51bfc9509013b0d85bc5a9d4f15b 2025-12-04T08:57:44.6757261Z * [new tag] trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 -> trunk/fdf863d5e1de3b2688c9511e96876e34581dbfd7 2025-12-04T08:57:44.6758796Z * [new tag] trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 -> trunk/fe0e65adfc0e7ca6e5f57e6ea8b16bd5cc967307 2025-12-04T08:57:44.6759819Z * [new tag] trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 -> trunk/fec710bf89173f5355468a7ce1afe9157c3d9009 2025-12-04T08:57:44.6760785Z * [new tag] trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 -> trunk/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:57:44.6761488Z * [new tag] v0.1.1 -> v0.1.1 2025-12-04T08:57:44.6762262Z * [new tag] v0.1.10 -> v0.1.10 2025-12-04T08:57:44.6763180Z * [new tag] v0.1.11 -> v0.1.11 2025-12-04T08:57:44.6764034Z * [new tag] v0.1.12 -> v0.1.12 2025-12-04T08:57:44.6764790Z * [new tag] v0.1.2 -> v0.1.2 2025-12-04T08:57:44.6765641Z * [new tag] v0.1.3 -> v0.1.3 2025-12-04T08:57:44.6766479Z * [new tag] v0.1.4 -> v0.1.4 2025-12-04T08:57:44.6767238Z * [new tag] v0.1.5 -> v0.1.5 2025-12-04T08:57:44.6768216Z * [new tag] v0.1.6 -> v0.1.6 2025-12-04T08:57:44.6768967Z * [new tag] v0.1.7 -> v0.1.7 2025-12-04T08:57:44.6769844Z * [new tag] v0.1.8 -> v0.1.8 2025-12-04T08:57:44.6770585Z * [new tag] v0.1.9 -> v0.1.9 2025-12-04T08:57:44.6771532Z * [new tag] v0.2.0 -> v0.2.0 2025-12-04T08:57:44.6772450Z * [new tag] v0.3.0 -> v0.3.0 2025-12-04T08:57:44.6773404Z * [new tag] v0.3.1 -> v0.3.1 2025-12-04T08:57:44.6774261Z * [new tag] v0.4.0 -> v0.4.0 2025-12-04T08:57:44.6775112Z * [new tag] v0.4.1 -> v0.4.1 2025-12-04T08:57:44.6775858Z * [new tag] v1.0.0 -> v1.0.0 2025-12-04T08:57:44.6776735Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-12-04T08:57:44.6777573Z * [new tag] v1.0.1 -> v1.0.1 2025-12-04T08:57:44.6778444Z * [new tag] v1.0rc0 -> v1.0rc0 2025-12-04T08:57:44.6779080Z * [new tag] v1.0rc1 -> v1.0rc1 2025-12-04T08:57:44.6780032Z * [new tag] v1.1.0 -> v1.1.0 2025-12-04T08:57:44.6780888Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-12-04T08:57:44.6781950Z * [new tag] v1.10.0 -> v1.10.0 2025-12-04T08:57:44.6782922Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-12-04T08:57:44.6783812Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-12-04T08:57:44.6784457Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-12-04T08:57:44.6785393Z * [new tag] v1.10.1 -> v1.10.1 2025-12-04T08:57:44.6786004Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-12-04T08:57:44.6786707Z * [new tag] v1.10.2 -> v1.10.2 2025-12-04T08:57:44.6787476Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-12-04T08:57:44.6788705Z * [new tag] v1.11.0 -> v1.11.0 2025-12-04T08:57:44.6789708Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-12-04T08:57:44.6790714Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-12-04T08:57:44.6792107Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-12-04T08:57:44.6793217Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-12-04T08:57:44.6794182Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-12-04T08:57:44.6794842Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-12-04T08:57:44.6795560Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-12-04T08:57:44.6796744Z * [new tag] v1.12.0 -> v1.12.0 2025-12-04T08:57:44.6797644Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-12-04T08:57:44.6798658Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-12-04T08:57:44.6799575Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-12-04T08:57:44.6800631Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-12-04T08:57:44.6801518Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-12-04T08:57:44.6802513Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-12-04T08:57:44.6803180Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-12-04T08:57:44.6803865Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-12-04T08:57:44.6804557Z * [new tag] v1.12.1 -> v1.12.1 2025-12-04T08:57:44.6805593Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-12-04T08:57:44.6806493Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-12-04T08:57:44.6807460Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-12-04T08:57:44.6808365Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-12-04T08:57:44.6809017Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-12-04T08:57:44.6809978Z * [new tag] v1.13.0 -> v1.13.0 2025-12-04T08:57:44.6810896Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-12-04T08:57:44.6811742Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-12-04T08:57:44.6812606Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-12-04T08:57:44.6813645Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-12-04T08:57:44.6814257Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-12-04T08:57:44.6814975Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-12-04T08:57:44.6815971Z * [new tag] v1.13.1 -> v1.13.1 2025-12-04T08:57:44.6816601Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-12-04T08:57:44.6817495Z * [new tag] v1.2.0 -> v1.2.0 2025-12-04T08:57:44.6818389Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-12-04T08:57:44.6819228Z * [new tag] v1.3.0 -> v1.3.0 2025-12-04T08:57:44.6820138Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-12-04T08:57:44.6820801Z * [new tag] v1.3.1 -> v1.3.1 2025-12-04T08:57:44.6821658Z * [new tag] v1.4.0 -> v1.4.0 2025-12-04T08:57:44.6822537Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-12-04T08:57:44.6823194Z * [new tag] v1.4.1 -> v1.4.1 2025-12-04T08:57:44.6825899Z * [new tag] v1.5.0 -> v1.5.0 2025-12-04T08:57:44.6827414Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-12-04T08:57:44.6828592Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-12-04T08:57:44.6829672Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-12-04T08:57:44.6830463Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-12-04T08:57:44.6831210Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-12-04T08:57:44.6832223Z * [new tag] v1.5.1 -> v1.5.1 2025-12-04T08:57:44.6832899Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-12-04T08:57:44.6833652Z * [new tag] v1.6.0 -> v1.6.0 2025-12-04T08:57:44.6834658Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-12-04T08:57:44.6835672Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-12-04T08:57:44.6836730Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-12-04T08:57:44.6837809Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-12-04T08:57:44.6838598Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-12-04T08:57:44.6839644Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-12-04T08:57:44.6840321Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-12-04T08:57:44.6841284Z * [new tag] v1.7.0 -> v1.7.0 2025-12-04T08:57:44.6842229Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-12-04T08:57:44.6843276Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-12-04T08:57:44.6844159Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-12-04T08:57:44.6844810Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-12-04T08:57:44.6845762Z * [new tag] v1.7.1 -> v1.7.1 2025-12-04T08:57:44.6846808Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-12-04T08:57:44.6847692Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-12-04T08:57:44.6848375Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-12-04T08:57:44.6849493Z * [new tag] v1.8.0 -> v1.8.0 2025-12-04T08:57:44.6850158Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-12-04T08:57:44.6851624Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-12-04T08:57:44.6852501Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-12-04T08:57:44.6853361Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-12-04T08:57:44.6853957Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-12-04T08:57:44.6854681Z * [new tag] v1.8.1 -> v1.8.1 2025-12-04T08:57:44.6855700Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-12-04T08:57:44.6856342Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-12-04T08:57:44.6857047Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-12-04T08:57:44.6858500Z * [new tag] v1.8.2 -> v1.8.2 2025-12-04T08:57:44.6859432Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-12-04T08:57:44.6860358Z * [new tag] v1.9.0 -> v1.9.0 2025-12-04T08:57:44.6861321Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-12-04T08:57:44.6862321Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-12-04T08:57:44.6863221Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-12-04T08:57:44.6863881Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-12-04T08:57:44.6864830Z * [new tag] v1.9.1 -> v1.9.1 2025-12-04T08:57:44.6865964Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-12-04T08:57:44.6866621Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-12-04T08:57:44.6867812Z * [new tag] v2.0.0 -> v2.0.0 2025-12-04T08:57:44.6868837Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-12-04T08:57:44.6869794Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-12-04T08:57:44.6870774Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-12-04T08:57:44.6871656Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-12-04T08:57:44.6872655Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-12-04T08:57:44.6873338Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-12-04T08:57:44.6874484Z * [new tag] v2.0.1 -> v2.0.1 2025-12-04T08:57:44.6875371Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-12-04T08:57:44.6875969Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-12-04T08:57:44.6876937Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-12-04T08:57:44.6877548Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-12-04T08:57:44.6879221Z * [new tag] v2.1.0 -> v2.1.0 2025-12-04T08:57:44.6880222Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-12-04T08:57:44.6881165Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-12-04T08:57:44.6882132Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-12-04T08:57:44.6883119Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-12-04T08:57:44.6884070Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-12-04T08:57:44.6884725Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-12-04T08:57:44.6885677Z * [new tag] v2.1.1 -> v2.1.1 2025-12-04T08:57:44.6886745Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-12-04T08:57:44.6887680Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-12-04T08:57:44.6888732Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-12-04T08:57:44.6889638Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-12-04T08:57:44.6890498Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-12-04T08:57:44.6891150Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-12-04T08:57:44.6892063Z * [new tag] v2.1.2 -> v2.1.2 2025-12-04T08:57:44.6893077Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-12-04T08:57:44.6894009Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-12-04T08:57:44.6894643Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-12-04T08:57:44.6895621Z * [new tag] v2.2.0 -> v2.2.0 2025-12-04T08:57:44.6896514Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-12-04T08:57:44.6897397Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-12-04T08:57:44.6898240Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-12-04T08:57:44.6899132Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-12-04T08:57:44.6899981Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-12-04T08:57:44.6900920Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-12-04T08:57:44.6901570Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-12-04T08:57:44.6902281Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-12-04T08:57:44.6903281Z * [new tag] v2.2.1 -> v2.2.1 2025-12-04T08:57:44.6904209Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-12-04T08:57:44.6904873Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-12-04T08:57:44.6905535Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-12-04T08:57:44.6906218Z * [new tag] v2.2.2 -> v2.2.2 2025-12-04T08:57:44.6907516Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-12-04T08:57:44.6908460Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-12-04T08:57:44.6909670Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-12-04T08:57:44.6910636Z * [new tag] v2.3.0 -> v2.3.0 2025-12-04T08:57:44.6911556Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-12-04T08:57:44.6912720Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-12-04T08:57:44.6913563Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-12-04T08:57:44.6914304Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-12-04T08:57:44.6915346Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-12-04T08:57:44.6916339Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-12-04T08:57:44.6917244Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-12-04T08:57:44.6918149Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-12-04T08:57:44.6918837Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-12-04T08:57:44.6919933Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-12-04T08:57:44.6920852Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-12-04T08:57:44.6921483Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-12-04T08:57:44.6922168Z * [new tag] v2.3.1 -> v2.3.1 2025-12-04T08:57:44.6923215Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-12-04T08:57:44.6924561Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-12-04T08:57:44.6925608Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-12-04T08:57:44.6926563Z * [new tag] v2.4.0 -> v2.4.0 2025-12-04T08:57:44.6927570Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-12-04T08:57:44.6928464Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-12-04T08:57:44.6929361Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-12-04T08:57:44.6930313Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-12-04T08:57:44.6931318Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-12-04T08:57:44.6932298Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-12-04T08:57:44.6933241Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-12-04T08:57:44.6934137Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-12-04T08:57:44.6935219Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-12-04T08:57:44.6936053Z * [new tag] v2.4.1 -> v2.4.1 2025-12-04T08:57:44.6937040Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-12-04T08:57:44.6938079Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-12-04T08:57:44.6939060Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-12-04T08:57:44.6939969Z * [new tag] v2.5.0 -> v2.5.0 2025-12-04T08:57:44.6940854Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-12-04T08:57:44.6941539Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-12-04T08:57:44.6942487Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-12-04T08:57:44.6943347Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-12-04T08:57:44.6944263Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-12-04T08:57:44.6945189Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-12-04T08:57:44.6946236Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-12-04T08:57:44.6947102Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-12-04T08:57:44.6948423Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-12-04T08:57:44.6949346Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-12-04T08:57:44.6950091Z * [new tag] v2.5.1 -> v2.5.1 2025-12-04T08:57:44.6950818Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-12-04T08:57:44.6951429Z * [new tag] v2.6.0 -> v2.6.0 2025-12-04T08:57:44.6952690Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-12-04T08:57:44.6953752Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-12-04T08:57:44.6954712Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-12-04T08:57:44.6955589Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-12-04T08:57:44.6956817Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-12-04T08:57:44.6957926Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-12-04T08:57:44.6958877Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-12-04T08:57:44.6960051Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-12-04T08:57:44.6961016Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-12-04T08:57:44.6962233Z * [new tag] v2.7.0 -> v2.7.0 2025-12-04T08:57:44.6963092Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-12-04T08:57:44.6963773Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-12-04T08:57:44.6964887Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-12-04T08:57:44.6965837Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-12-04T08:57:44.6966757Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-12-04T08:57:44.6967676Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-12-04T08:57:44.6968661Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-12-04T08:57:44.6969621Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-12-04T08:57:44.6971022Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-12-04T08:57:44.6972012Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-12-04T08:57:44.6972652Z * [new tag] v2.7.1 -> v2.7.1 2025-12-04T08:57:44.6973789Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-12-04T08:57:44.6974715Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-12-04T08:57:44.6975766Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-12-04T08:57:44.6976759Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-12-04T08:57:44.6977723Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-12-04T08:57:44.6978401Z * [new tag] v2.8.0 -> v2.8.0 2025-12-04T08:57:44.6979376Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-12-04T08:57:44.6980325Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-12-04T08:57:44.6981362Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-12-04T08:57:44.6982413Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-12-04T08:57:44.6983382Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-12-04T08:57:44.6984345Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-12-04T08:57:44.6985340Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-12-04T08:57:44.6986236Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-12-04T08:57:44.6987292Z * [new tag] v2.9.0 -> v2.9.0 2025-12-04T08:57:44.6988561Z * [new tag] v2.9.0-rc1 -> v2.9.0-rc1 2025-12-04T08:57:44.6989544Z * [new tag] v2.9.0-rc10 -> v2.9.0-rc10 2025-12-04T08:57:44.6990594Z * [new tag] v2.9.0-rc11 -> v2.9.0-rc11 2025-12-04T08:57:44.6991913Z * [new tag] v2.9.0-rc2 -> v2.9.0-rc2 2025-12-04T08:57:44.6992812Z * [new tag] v2.9.0-rc3 -> v2.9.0-rc3 2025-12-04T08:57:44.6993881Z * [new tag] v2.9.0-rc4 -> v2.9.0-rc4 2025-12-04T08:57:44.6994818Z * [new tag] v2.9.0-rc5 -> v2.9.0-rc5 2025-12-04T08:57:44.6996091Z * [new tag] v2.9.0-rc6 -> v2.9.0-rc6 2025-12-04T08:57:44.6997053Z * [new tag] v2.9.0-rc7 -> v2.9.0-rc7 2025-12-04T08:57:44.6998281Z * [new tag] v2.9.0-rc8 -> v2.9.0-rc8 2025-12-04T08:57:44.6998926Z * [new tag] v2.9.0-rc9 -> v2.9.0-rc9 2025-12-04T08:57:44.6999675Z * [new tag] v2.9.1 -> v2.9.1 2025-12-04T08:57:44.7000961Z * [new tag] v2.9.1-rc1 -> v2.9.1-rc1 2025-12-04T08:57:44.7001946Z * [new tag] v2.9.1-rc2 -> v2.9.1-rc2 2025-12-04T08:57:44.7003183Z * [new tag] viable/strict/1759343184 -> viable/strict/1759343184 2025-12-04T08:57:44.7004093Z * [new tag] viable/strict/1759346540 -> viable/strict/1759346540 2025-12-04T08:57:44.7004883Z * [new tag] viable/strict/1759348181 -> viable/strict/1759348181 2025-12-04T08:57:44.7005814Z * [new tag] viable/strict/1759350324 -> viable/strict/1759350324 2025-12-04T08:57:44.7006635Z * [new tag] viable/strict/1759351793 -> viable/strict/1759351793 2025-12-04T08:57:44.7007563Z * [new tag] viable/strict/1759353844 -> viable/strict/1759353844 2025-12-04T08:57:44.7008358Z * [new tag] viable/strict/1759355374 -> viable/strict/1759355374 2025-12-04T08:57:44.7009276Z * [new tag] viable/strict/1759357472 -> viable/strict/1759357472 2025-12-04T08:57:44.7010018Z * [new tag] viable/strict/1759361002 -> viable/strict/1759361002 2025-12-04T08:57:44.7011227Z * [new tag] viable/strict/1759362585 -> viable/strict/1759362585 2025-12-04T08:57:44.7012326Z * [new tag] viable/strict/1759365359 -> viable/strict/1759365359 2025-12-04T08:57:44.7013263Z * [new tag] viable/strict/1759370089 -> viable/strict/1759370089 2025-12-04T08:57:44.7014622Z * [new tag] viable/strict/1759377554 -> viable/strict/1759377554 2025-12-04T08:57:44.7015528Z * [new tag] viable/strict/1759379133 -> viable/strict/1759379133 2025-12-04T08:57:44.7016435Z * [new tag] viable/strict/1759389871 -> viable/strict/1759389871 2025-12-04T08:57:44.7017338Z * [new tag] viable/strict/1759393562 -> viable/strict/1759393562 2025-12-04T08:57:44.7018310Z * [new tag] viable/strict/1759395076 -> viable/strict/1759395076 2025-12-04T08:57:44.7019254Z * [new tag] viable/strict/1759398579 -> viable/strict/1759398579 2025-12-04T08:57:44.7020136Z * [new tag] viable/strict/1759404142 -> viable/strict/1759404142 2025-12-04T08:57:44.7020971Z * [new tag] viable/strict/1759405773 -> viable/strict/1759405773 2025-12-04T08:57:44.7021897Z * [new tag] viable/strict/1759408041 -> viable/strict/1759408041 2025-12-04T08:57:44.7022728Z * [new tag] viable/strict/1759411593 -> viable/strict/1759411593 2025-12-04T08:57:44.7023812Z * [new tag] viable/strict/1759427395 -> viable/strict/1759427395 2025-12-04T08:57:44.7025022Z * [new tag] viable/strict/1759434582 -> viable/strict/1759434582 2025-12-04T08:57:44.7025958Z * [new tag] viable/strict/1759436720 -> viable/strict/1759436720 2025-12-04T08:57:44.7026800Z * [new tag] viable/strict/1759440219 -> viable/strict/1759440219 2025-12-04T08:57:44.7027905Z * [new tag] viable/strict/1759441948 -> viable/strict/1759441948 2025-12-04T08:57:44.7028862Z * [new tag] viable/strict/1759443860 -> viable/strict/1759443860 2025-12-04T08:57:44.7029791Z * [new tag] viable/strict/1759445377 -> viable/strict/1759445377 2025-12-04T08:57:44.7030866Z * [new tag] viable/strict/1759447415 -> viable/strict/1759447415 2025-12-04T08:57:44.7031793Z * [new tag] viable/strict/1759451750 -> viable/strict/1759451750 2025-12-04T08:57:44.7032751Z * [new tag] viable/strict/1759453910 -> viable/strict/1759453910 2025-12-04T08:57:44.7033694Z * [new tag] viable/strict/1759456483 -> viable/strict/1759456483 2025-12-04T08:57:44.7034709Z * [new tag] viable/strict/1759459279 -> viable/strict/1759459279 2025-12-04T08:57:44.7035632Z * [new tag] viable/strict/1759460742 -> viable/strict/1759460742 2025-12-04T08:57:44.7036539Z * [new tag] viable/strict/1759462025 -> viable/strict/1759462025 2025-12-04T08:57:44.7037488Z * [new tag] viable/strict/1759469086 -> viable/strict/1759469086 2025-12-04T08:57:44.7038411Z * [new tag] viable/strict/1759470581 -> viable/strict/1759470581 2025-12-04T08:57:44.7039475Z * [new tag] viable/strict/1759472786 -> viable/strict/1759472786 2025-12-04T08:57:44.7040373Z * [new tag] viable/strict/1759476294 -> viable/strict/1759476294 2025-12-04T08:57:44.7041281Z * [new tag] viable/strict/1759479963 -> viable/strict/1759479963 2025-12-04T08:57:44.7042191Z * [new tag] viable/strict/1759492177 -> viable/strict/1759492177 2025-12-04T08:57:44.7043083Z * [new tag] viable/strict/1759519278 -> viable/strict/1759519278 2025-12-04T08:57:44.7043978Z * [new tag] viable/strict/1759524580 -> viable/strict/1759524580 2025-12-04T08:57:44.7044931Z * [new tag] viable/strict/1759528193 -> viable/strict/1759528193 2025-12-04T08:57:44.7045990Z * [new tag] viable/strict/1759533797 -> viable/strict/1759533797 2025-12-04T08:57:44.7046904Z * [new tag] viable/strict/1759542780 -> viable/strict/1759542780 2025-12-04T08:57:44.7047817Z * [new tag] viable/strict/1759549779 -> viable/strict/1759549779 2025-12-04T08:57:44.7048742Z * [new tag] viable/strict/1759555455 -> viable/strict/1759555455 2025-12-04T08:57:44.7049642Z * [new tag] viable/strict/1759559176 -> viable/strict/1759559176 2025-12-04T08:57:44.7050704Z * [new tag] viable/strict/1759560629 -> viable/strict/1759560629 2025-12-04T08:57:44.7051614Z * [new tag] viable/strict/1759569848 -> viable/strict/1759569848 2025-12-04T08:57:44.7052752Z * [new tag] viable/strict/1759571382 -> viable/strict/1759571382 2025-12-04T08:57:44.7053580Z * [new tag] viable/strict/1759573474 -> viable/strict/1759573474 2025-12-04T08:57:44.7054509Z * [new tag] viable/strict/1759618187 -> viable/strict/1759618187 2025-12-04T08:57:44.7055320Z * [new tag] viable/strict/1759626742 -> viable/strict/1759626742 2025-12-04T08:57:44.7056305Z * [new tag] viable/strict/1759632427 -> viable/strict/1759632427 2025-12-04T08:57:44.7057131Z * [new tag] viable/strict/1759634971 -> viable/strict/1759634971 2025-12-04T08:57:44.7058178Z * [new tag] viable/strict/1759661382 -> viable/strict/1759661382 2025-12-04T08:57:44.7059087Z * [new tag] viable/strict/1759663294 -> viable/strict/1759663294 2025-12-04T08:57:44.7059888Z * [new tag] viable/strict/1759708178 -> viable/strict/1759708178 2025-12-04T08:57:44.7060841Z * [new tag] viable/strict/1759715695 -> viable/strict/1759715695 2025-12-04T08:57:44.7061817Z * [new tag] viable/strict/1759728293 -> viable/strict/1759728293 2025-12-04T08:57:44.7062587Z * [new tag] viable/strict/1759735513 -> viable/strict/1759735513 2025-12-04T08:57:44.7063615Z * [new tag] viable/strict/1759739177 -> viable/strict/1759739177 2025-12-04T08:57:44.7064441Z * [new tag] viable/strict/1759758635 -> viable/strict/1759758635 2025-12-04T08:57:44.7065409Z * [new tag] viable/strict/1759765784 -> viable/strict/1759765784 2025-12-04T08:57:44.7066220Z * [new tag] viable/strict/1759767948 -> viable/strict/1759767948 2025-12-04T08:57:44.7067196Z * [new tag] viable/strict/1759771461 -> viable/strict/1759771461 2025-12-04T08:57:44.7068285Z * [new tag] viable/strict/1759776706 -> viable/strict/1759776706 2025-12-04T08:57:44.7069307Z * [new tag] viable/strict/1759782317 -> viable/strict/1759782317 2025-12-04T08:57:44.7070293Z * [new tag] viable/strict/1759783777 -> viable/strict/1759783777 2025-12-04T08:57:44.7071263Z * [new tag] viable/strict/1759785815 -> viable/strict/1759785815 2025-12-04T08:57:44.7072194Z * [new tag] viable/strict/1759789459 -> viable/strict/1759789459 2025-12-04T08:57:44.7073126Z * [new tag] viable/strict/1759790974 -> viable/strict/1759790974 2025-12-04T08:57:44.7073872Z * [new tag] viable/strict/1759794583 -> viable/strict/1759794583 2025-12-04T08:57:44.7075316Z * [new tag] viable/strict/1759797408 -> viable/strict/1759797408 2025-12-04T08:57:44.7076256Z * [new tag] viable/strict/1759799518 -> viable/strict/1759799518 2025-12-04T08:57:44.7077189Z * [new tag] viable/strict/1759804909 -> viable/strict/1759804909 2025-12-04T08:57:44.7078148Z * [new tag] viable/strict/1759807643 -> viable/strict/1759807643 2025-12-04T08:57:44.7079087Z * [new tag] viable/strict/1759809089 -> viable/strict/1759809089 2025-12-04T08:57:44.7080113Z * [new tag] viable/strict/1759811145 -> viable/strict/1759811145 2025-12-04T08:57:44.7081028Z * [new tag] viable/strict/1759812581 -> viable/strict/1759812581 2025-12-04T08:57:44.7081936Z * [new tag] viable/strict/1759814683 -> viable/strict/1759814683 2025-12-04T08:57:44.7082840Z * [new tag] viable/strict/1759821889 -> viable/strict/1759821889 2025-12-04T08:57:44.7083822Z * [new tag] viable/strict/1759823376 -> viable/strict/1759823376 2025-12-04T08:57:44.7084637Z * [new tag] viable/strict/1759827107 -> viable/strict/1759827107 2025-12-04T08:57:44.7085557Z * [new tag] viable/strict/1759830577 -> viable/strict/1759830577 2025-12-04T08:57:44.7086528Z * [new tag] viable/strict/1759832720 -> viable/strict/1759832720 2025-12-04T08:57:44.7087431Z * [new tag] viable/strict/1759842063 -> viable/strict/1759842063 2025-12-04T08:57:44.7088265Z * [new tag] viable/strict/1759847121 -> viable/strict/1759847121 2025-12-04T08:57:44.7089529Z * [new tag] viable/strict/1759850721 -> viable/strict/1759850721 2025-12-04T08:57:44.7090590Z * [new tag] viable/strict/1759857870 -> viable/strict/1759857870 2025-12-04T08:57:44.7091528Z * [new tag] viable/strict/1759863143 -> viable/strict/1759863143 2025-12-04T08:57:44.7092418Z * [new tag] viable/strict/1759875874 -> viable/strict/1759875874 2025-12-04T08:57:44.7093113Z * [new tag] viable/strict/1759877385 -> viable/strict/1759877385 2025-12-04T08:57:44.7094076Z * [new tag] viable/strict/1759883801 -> viable/strict/1759883801 2025-12-04T08:57:44.7094988Z * [new tag] viable/strict/1759885922 -> viable/strict/1759885922 2025-12-04T08:57:44.7095974Z * [new tag] viable/strict/1759888488 -> viable/strict/1759888488 2025-12-04T08:57:44.7096782Z * [new tag] viable/strict/1759895471 -> viable/strict/1759895471 2025-12-04T08:57:44.7097793Z * [new tag] viable/strict/1759904803 -> viable/strict/1759904803 2025-12-04T08:57:44.7098831Z * [new tag] viable/strict/1759908300 -> viable/strict/1759908300 2025-12-04T08:57:44.7099778Z * [new tag] viable/strict/1759915520 -> viable/strict/1759915520 2025-12-04T08:57:44.7100662Z * [new tag] viable/strict/1759916978 -> viable/strict/1759916978 2025-12-04T08:57:44.7101385Z * [new tag] viable/strict/1759930024 -> viable/strict/1759930024 2025-12-04T08:57:44.7102400Z * [new tag] viable/strict/1759948122 -> viable/strict/1759948122 2025-12-04T08:57:44.7103342Z * [new tag] viable/strict/1759952983 -> viable/strict/1759952983 2025-12-04T08:57:44.7104288Z * [new tag] viable/strict/1759955121 -> viable/strict/1759955121 2025-12-04T08:57:44.7105100Z * [new tag] viable/strict/1759962298 -> viable/strict/1759962298 2025-12-04T08:57:44.7106082Z * [new tag] viable/strict/1759965837 -> viable/strict/1759965837 2025-12-04T08:57:44.7107005Z * [new tag] viable/strict/1759970213 -> viable/strict/1759970213 2025-12-04T08:57:44.7108278Z * [new tag] viable/strict/1759974894 -> viable/strict/1759974894 2025-12-04T08:57:44.7109203Z * [new tag] viable/strict/1759977763 -> viable/strict/1759977763 2025-12-04T08:57:44.7110167Z * [new tag] viable/strict/1759979241 -> viable/strict/1759979241 2025-12-04T08:57:44.7111134Z * [new tag] viable/strict/1759985417 -> viable/strict/1759985417 2025-12-04T08:57:44.7112081Z * [new tag] viable/strict/1759987490 -> viable/strict/1759987490 2025-12-04T08:57:44.7113058Z * [new tag] viable/strict/1759996180 -> viable/strict/1759996180 2025-12-04T08:57:44.7113968Z * [new tag] viable/strict/1760065682 -> viable/strict/1760065682 2025-12-04T08:57:44.7114911Z * [new tag] viable/strict/1760066894 -> viable/strict/1760066894 2025-12-04T08:57:44.7115849Z * [new tag] viable/strict/1760070345 -> viable/strict/1760070345 2025-12-04T08:57:44.7116771Z * [new tag] viable/strict/1760089782 -> viable/strict/1760089782 2025-12-04T08:57:44.7117709Z * [new tag] viable/strict/1760091921 -> viable/strict/1760091921 2025-12-04T08:57:44.7118715Z * [new tag] viable/strict/1760127924 -> viable/strict/1760127924 2025-12-04T08:57:44.7119794Z * [new tag] viable/strict/1760129489 -> viable/strict/1760129489 2025-12-04T08:57:44.7120898Z * [new tag] viable/strict/1760132980 -> viable/strict/1760132980 2025-12-04T08:57:44.7121887Z * [new tag] viable/strict/1760135060 -> viable/strict/1760135060 2025-12-04T08:57:44.7122787Z * [new tag] viable/strict/1760215782 -> viable/strict/1760215782 2025-12-04T08:57:44.7123903Z * [new tag] viable/strict/1760273849 -> viable/strict/1760273849 2025-12-04T08:57:44.7125111Z * [new tag] viable/strict/1760275517 -> viable/strict/1760275517 2025-12-04T08:57:44.7126032Z * [new tag] viable/strict/1760276979 -> viable/strict/1760276979 2025-12-04T08:57:44.7126972Z * [new tag] viable/strict/1760279007 -> viable/strict/1760279007 2025-12-04T08:57:44.7127704Z * [new tag] viable/strict/1760286328 -> viable/strict/1760286328 2025-12-04T08:57:44.7128475Z * [new tag] viable/strict/1760493304 -> viable/strict/1760493304 2025-12-04T08:57:44.7129484Z * [new tag] viable/strict/1760496298 -> viable/strict/1760496298 2025-12-04T08:57:44.7130515Z * [new tag] viable/strict/1760518396 -> viable/strict/1760518396 2025-12-04T08:57:44.7131313Z * [new tag] viable/strict/1760534864 -> viable/strict/1760534864 2025-12-04T08:57:44.7132320Z * [new tag] viable/strict/1760549062 -> viable/strict/1760549062 2025-12-04T08:57:44.7133412Z * [new tag] viable/strict/1760552799 -> viable/strict/1760552799 2025-12-04T08:57:44.7134322Z * [new tag] viable/strict/1760554355 -> viable/strict/1760554355 2025-12-04T08:57:44.7135313Z * [new tag] viable/strict/1760556275 -> viable/strict/1760556275 2025-12-04T08:57:44.7136802Z * [new tag] viable/strict/1760564979 -> viable/strict/1760564979 2025-12-04T08:57:44.7137821Z * [new tag] viable/strict/1760567049 -> viable/strict/1760567049 2025-12-04T08:57:44.7139168Z * [new tag] viable/strict/1760568585 -> viable/strict/1760568585 2025-12-04T08:57:44.7140066Z * [new tag] viable/strict/1760570630 -> viable/strict/1760570630 2025-12-04T08:57:44.7140990Z * [new tag] viable/strict/1760572180 -> viable/strict/1760572180 2025-12-04T08:57:44.7141891Z * [new tag] viable/strict/1760575094 -> viable/strict/1760575094 2025-12-04T08:57:44.7142939Z * [new tag] viable/strict/1760579709 -> viable/strict/1760579709 2025-12-04T08:57:44.7144290Z * [new tag] viable/strict/1760582614 -> viable/strict/1760582614 2025-12-04T08:57:44.7145269Z * [new tag] viable/strict/1760586815 -> viable/strict/1760586815 2025-12-04T08:57:44.7145991Z * [new tag] viable/strict/1760588829 -> viable/strict/1760588829 2025-12-04T08:57:44.7147037Z * [new tag] viable/strict/1760590200 -> viable/strict/1760590200 2025-12-04T08:57:44.7148383Z * [new tag] viable/strict/1760592311 -> viable/strict/1760592311 2025-12-04T08:57:44.7149315Z * [new tag] viable/strict/1760619733 -> viable/strict/1760619733 2025-12-04T08:57:44.7150048Z * [new tag] viable/strict/1760628335 -> viable/strict/1760628335 2025-12-04T08:57:44.7151016Z * [new tag] viable/strict/1760635490 -> viable/strict/1760635490 2025-12-04T08:57:44.7151858Z * [new tag] viable/strict/1760640743 -> viable/strict/1760640743 2025-12-04T08:57:44.7152985Z * [new tag] viable/strict/1760642528 -> viable/strict/1760642528 2025-12-04T08:57:44.7153812Z * [new tag] viable/strict/1760646330 -> viable/strict/1760646330 2025-12-04T08:57:44.7154827Z * [new tag] viable/strict/1760666101 -> viable/strict/1760666101 2025-12-04T08:57:44.7155774Z * [new tag] viable/strict/1760668990 -> viable/strict/1760668990 2025-12-04T08:57:44.7156735Z * [new tag] viable/strict/1760670600 -> viable/strict/1760670600 2025-12-04T08:57:44.7157589Z * [new tag] viable/strict/1760671704 -> viable/strict/1760671704 2025-12-04T08:57:44.7158573Z * [new tag] viable/strict/1760673121 -> viable/strict/1760673121 2025-12-04T08:57:44.7159620Z * [new tag] viable/strict/1760675352 -> viable/strict/1760675352 2025-12-04T08:57:44.7160536Z * [new tag] viable/strict/1760696731 -> viable/strict/1760696731 2025-12-04T08:57:44.7162992Z * [new tag] viable/strict/1760723515 -> viable/strict/1760723515 2025-12-04T08:57:44.7163892Z * [new tag] viable/strict/1760727234 -> viable/strict/1760727234 2025-12-04T08:57:44.7164862Z * [new tag] viable/strict/1760730578 -> viable/strict/1760730578 2025-12-04T08:57:44.7165805Z * [new tag] viable/strict/1760732726 -> viable/strict/1760732726 2025-12-04T08:57:44.7166702Z * [new tag] viable/strict/1760734180 -> viable/strict/1760734180 2025-12-04T08:57:44.7167781Z * [new tag] viable/strict/1760736251 -> viable/strict/1760736251 2025-12-04T08:57:44.7168510Z * [new tag] viable/strict/1760737772 -> viable/strict/1760737772 2025-12-04T08:57:44.7169507Z * [new tag] viable/strict/1760758005 -> viable/strict/1760758005 2025-12-04T08:57:44.7170321Z * [new tag] viable/strict/1760761532 -> viable/strict/1760761532 2025-12-04T08:57:44.7171297Z * [new tag] viable/strict/1760802581 -> viable/strict/1760802581 2025-12-04T08:57:44.7172248Z * [new tag] viable/strict/1760827772 -> viable/strict/1760827772 2025-12-04T08:57:44.7173160Z * [new tag] viable/strict/1760834524 -> viable/strict/1760834524 2025-12-04T08:57:44.7174170Z * [new tag] viable/strict/1760845009 -> viable/strict/1760845009 2025-12-04T08:57:44.7175099Z * [new tag] viable/strict/1760876836 -> viable/strict/1760876836 2025-12-04T08:57:44.7176023Z * [new tag] viable/strict/1760880329 -> viable/strict/1760880329 2025-12-04T08:57:44.7176987Z * [new tag] viable/strict/1760888987 -> viable/strict/1760888987 2025-12-04T08:57:44.7177737Z * [new tag] viable/strict/1760912664 -> viable/strict/1760912664 2025-12-04T08:57:44.7178729Z * [new tag] viable/strict/1760925321 -> viable/strict/1760925321 2025-12-04T08:57:44.7179545Z * [new tag] viable/strict/1760931488 -> viable/strict/1760931488 2025-12-04T08:57:44.7180491Z * [new tag] viable/strict/1760932693 -> viable/strict/1760932693 2025-12-04T08:57:44.7181430Z * [new tag] viable/strict/1761004184 -> viable/strict/1761004184 2025-12-04T08:57:44.7182329Z * [new tag] viable/strict/1761014748 -> viable/strict/1761014748 2025-12-04T08:57:44.7183237Z * [new tag] viable/strict/1761017491 -> viable/strict/1761017491 2025-12-04T08:57:44.7184301Z * [new tag] viable/strict/1761018806 -> viable/strict/1761018806 2025-12-04T08:57:44.7185294Z * [new tag] viable/strict/1761020754 -> viable/strict/1761020754 2025-12-04T08:57:44.7186188Z * [new tag] viable/strict/1761024303 -> viable/strict/1761024303 2025-12-04T08:57:44.7187023Z * [new tag] viable/strict/1761029582 -> viable/strict/1761029582 2025-12-04T08:57:44.7188308Z * [new tag] viable/strict/1761031535 -> viable/strict/1761031535 2025-12-04T08:57:44.7189234Z * [new tag] viable/strict/1761035196 -> viable/strict/1761035196 2025-12-04T08:57:44.7190281Z * [new tag] viable/strict/1761045825 -> viable/strict/1761045825 2025-12-04T08:57:44.7191250Z * [new tag] viable/strict/1761054796 -> viable/strict/1761054796 2025-12-04T08:57:44.7192204Z * [new tag] viable/strict/1761060314 -> viable/strict/1761060314 2025-12-04T08:57:44.7193145Z * [new tag] viable/strict/1761071198 -> viable/strict/1761071198 2025-12-04T08:57:44.7194165Z * [new tag] viable/strict/1761074628 -> viable/strict/1761074628 2025-12-04T08:57:44.7195131Z * [new tag] viable/strict/1761078351 -> viable/strict/1761078351 2025-12-04T08:57:44.7196061Z * [new tag] viable/strict/1761079822 -> viable/strict/1761079822 2025-12-04T08:57:44.7196997Z * [new tag] viable/strict/1761081873 -> viable/strict/1761081873 2025-12-04T08:57:44.7197954Z * [new tag] viable/strict/1761083392 -> viable/strict/1761083392 2025-12-04T08:57:44.7198942Z * [new tag] viable/strict/1761085465 -> viable/strict/1761085465 2025-12-04T08:57:44.7200445Z * [new tag] viable/strict/1761089099 -> viable/strict/1761089099 2025-12-04T08:57:44.7201337Z * [new tag] viable/strict/1761095535 -> viable/strict/1761095535 2025-12-04T08:57:44.7202330Z * [new tag] viable/strict/1761098119 -> viable/strict/1761098119 2025-12-04T08:57:44.7203633Z * [new tag] viable/strict/1761101330 -> viable/strict/1761101330 2025-12-04T08:57:44.7204583Z * [new tag] viable/strict/1761114425 -> viable/strict/1761114425 2025-12-04T08:57:44.7205487Z * [new tag] viable/strict/1761116036 -> viable/strict/1761116036 2025-12-04T08:57:44.7206412Z * [new tag] viable/strict/1761119379 -> viable/strict/1761119379 2025-12-04T08:57:44.7207309Z * [new tag] viable/strict/1761121601 -> viable/strict/1761121601 2025-12-04T08:57:44.7208238Z * [new tag] viable/strict/1761123234 -> viable/strict/1761123234 2025-12-04T08:57:44.7209075Z * [new tag] viable/strict/1761126621 -> viable/strict/1761126621 2025-12-04T08:57:44.7209993Z * [new tag] viable/strict/1761132259 -> viable/strict/1761132259 2025-12-04T08:57:44.7210996Z * [new tag] viable/strict/1761146746 -> viable/strict/1761146746 2025-12-04T08:57:44.7211934Z * [new tag] viable/strict/1761164752 -> viable/strict/1761164752 2025-12-04T08:57:44.7212733Z * [new tag] viable/strict/1761166198 -> viable/strict/1761166198 2025-12-04T08:57:44.7213716Z * [new tag] viable/strict/1761175424 -> viable/strict/1761175424 2025-12-04T08:57:44.7214632Z * [new tag] viable/strict/1761176983 -> viable/strict/1761176983 2025-12-04T08:57:44.7215815Z * [new tag] viable/strict/1761179891 -> viable/strict/1761179891 2025-12-04T08:57:44.7216713Z * [new tag] viable/strict/1761181930 -> viable/strict/1761181930 2025-12-04T08:57:44.7217677Z * [new tag] viable/strict/1761184516 -> viable/strict/1761184516 2025-12-04T08:57:44.7218661Z * [new tag] viable/strict/1761190179 -> viable/strict/1761190179 2025-12-04T08:57:44.7219576Z * [new tag] viable/strict/1761193558 -> viable/strict/1761193558 2025-12-04T08:57:44.7220468Z * [new tag] viable/strict/1761207990 -> viable/strict/1761207990 2025-12-04T08:57:44.7221374Z * [new tag] viable/strict/1761229539 -> viable/strict/1761229539 2025-12-04T08:57:44.7222563Z * [new tag] viable/strict/1761244031 -> viable/strict/1761244031 2025-12-04T08:57:44.7223385Z * [new tag] viable/strict/1761248986 -> viable/strict/1761248986 2025-12-04T08:57:44.7224935Z * [new tag] viable/strict/1761259791 -> viable/strict/1761259791 2025-12-04T08:57:44.7226137Z * [new tag] viable/strict/1761266139 -> viable/strict/1761266139 2025-12-04T08:57:44.7227304Z * [new tag] viable/strict/1761268316 -> viable/strict/1761268316 2025-12-04T08:57:44.7228334Z * [new tag] viable/strict/1761273805 -> viable/strict/1761273805 2025-12-04T08:57:44.7229240Z * [new tag] viable/strict/1761275261 -> viable/strict/1761275261 2025-12-04T08:57:44.7230229Z * [new tag] viable/strict/1761277913 -> viable/strict/1761277913 2025-12-04T08:57:44.7231297Z * [new tag] viable/strict/1761290701 -> viable/strict/1761290701 2025-12-04T08:57:44.7232295Z * [new tag] viable/strict/1761294396 -> viable/strict/1761294396 2025-12-04T08:57:44.7233224Z * [new tag] viable/strict/1761303047 -> viable/strict/1761303047 2025-12-04T08:57:44.7234145Z * [new tag] viable/strict/1761335388 -> viable/strict/1761335388 2025-12-04T08:57:44.7235148Z * [new tag] viable/strict/1761337551 -> viable/strict/1761337551 2025-12-04T08:57:44.7235976Z * [new tag] viable/strict/1761339007 -> viable/strict/1761339007 2025-12-04T08:57:44.7236997Z * [new tag] viable/strict/1761341050 -> viable/strict/1761341050 2025-12-04T08:57:44.7238139Z * [new tag] viable/strict/1761346188 -> viable/strict/1761346188 2025-12-04T08:57:44.7239170Z * [new tag] viable/strict/1761349792 -> viable/strict/1761349792 2025-12-04T08:57:44.7240117Z * [new tag] viable/strict/1761352620 -> viable/strict/1761352620 2025-12-04T08:57:44.7241017Z * [new tag] viable/strict/1761354730 -> viable/strict/1761354730 2025-12-04T08:57:44.7241972Z * [new tag] viable/strict/1761357298 -> viable/strict/1761357298 2025-12-04T08:57:44.7242908Z * [new tag] viable/strict/1761360201 -> viable/strict/1761360201 2025-12-04T08:57:44.7243855Z * [new tag] viable/strict/1761361753 -> viable/strict/1761361753 2025-12-04T08:57:44.7244749Z * [new tag] viable/strict/1761364351 -> viable/strict/1761364351 2025-12-04T08:57:44.7245695Z * [new tag] viable/strict/1761366338 -> viable/strict/1761366338 2025-12-04T08:57:44.7246929Z * [new tag] viable/strict/1761367802 -> viable/strict/1761367802 2025-12-04T08:57:44.7247986Z * [new tag] viable/strict/1761369889 -> viable/strict/1761369889 2025-12-04T08:57:44.7248943Z * [new tag] viable/strict/1761371385 -> viable/strict/1761371385 2025-12-04T08:57:44.7249870Z * [new tag] viable/strict/1761373581 -> viable/strict/1761373581 2025-12-04T08:57:44.7250930Z * [new tag] viable/strict/1761375054 -> viable/strict/1761375054 2025-12-04T08:57:44.7251866Z * [new tag] viable/strict/1761421785 -> viable/strict/1761421785 2025-12-04T08:57:44.7252859Z * [new tag] viable/strict/1761434614 -> viable/strict/1761434614 2025-12-04T08:57:44.7254136Z * [new tag] viable/strict/1761439254 -> viable/strict/1761439254 2025-12-04T08:57:44.7255132Z * [new tag] viable/strict/1761454187 -> viable/strict/1761454187 2025-12-04T08:57:44.7256135Z * [new tag] viable/strict/1761459991 -> viable/strict/1761459991 2025-12-04T08:57:44.7257197Z * [new tag] viable/strict/1761470668 -> viable/strict/1761470668 2025-12-04T08:57:44.7258543Z * [new tag] viable/strict/1761472188 -> viable/strict/1761472188 2025-12-04T08:57:44.7259488Z * [new tag] viable/strict/1761503178 -> viable/strict/1761503178 2025-12-04T08:57:44.7260415Z * [new tag] viable/strict/1761517492 -> viable/strict/1761517492 2025-12-04T08:57:44.7261330Z * [new tag] viable/strict/1761518981 -> viable/strict/1761518981 2025-12-04T08:57:44.7262308Z * [new tag] viable/strict/1761533609 -> viable/strict/1761533609 2025-12-04T08:57:44.7263070Z * [new tag] viable/strict/1761546438 -> viable/strict/1761546438 2025-12-04T08:57:44.7264585Z * [new tag] viable/strict/1761548133 -> viable/strict/1761548133 2025-12-04T08:57:44.7265775Z * [new tag] viable/strict/1761555186 -> viable/strict/1761555186 2025-12-04T08:57:44.7266807Z * [new tag] viable/strict/1761557178 -> viable/strict/1761557178 2025-12-04T08:57:44.7268033Z * [new tag] viable/strict/1761560772 -> viable/strict/1761560772 2025-12-04T08:57:44.7269059Z * [new tag] viable/strict/1761562266 -> viable/strict/1761562266 2025-12-04T08:57:44.7270112Z * [new tag] viable/strict/1761564260 -> viable/strict/1761564260 2025-12-04T08:57:44.7271057Z * [new tag] viable/strict/1761568072 -> viable/strict/1761568072 2025-12-04T08:57:44.7271988Z * [new tag] viable/strict/1761571683 -> viable/strict/1761571683 2025-12-04T08:57:44.7272752Z * [new tag] viable/strict/1761580199 -> viable/strict/1761580199 2025-12-04T08:57:44.7273747Z * [new tag] viable/strict/1761587383 -> viable/strict/1761587383 2025-12-04T08:57:44.7274860Z * [new tag] viable/strict/1761591165 -> viable/strict/1761591165 2025-12-04T08:57:44.7275671Z * [new tag] viable/strict/1761594575 -> viable/strict/1761594575 2025-12-04T08:57:44.7276717Z * [new tag] viable/strict/1761596710 -> viable/strict/1761596710 2025-12-04T08:57:44.7277687Z * [new tag] viable/strict/1761598189 -> viable/strict/1761598189 2025-12-04T08:57:44.7278652Z * [new tag] viable/strict/1761600254 -> viable/strict/1761600254 2025-12-04T08:57:44.7279726Z * [new tag] viable/strict/1761603879 -> viable/strict/1761603879 2025-12-04T08:57:44.7280808Z * [new tag] viable/strict/1761605429 -> viable/strict/1761605429 2025-12-04T08:57:44.7281865Z * [new tag] viable/strict/1761607468 -> viable/strict/1761607468 2025-12-04T08:57:44.7282820Z * [new tag] viable/strict/1761608983 -> viable/strict/1761608983 2025-12-04T08:57:44.7283746Z * [new tag] viable/strict/1761611846 -> viable/strict/1761611846 2025-12-04T08:57:44.7284720Z * [new tag] viable/strict/1761613922 -> viable/strict/1761613922 2025-12-04T08:57:44.7285440Z * [new tag] viable/strict/1761616504 -> viable/strict/1761616504 2025-12-04T08:57:44.7286250Z * [new tag] viable/strict/1761619599 -> viable/strict/1761619599 2025-12-04T08:57:44.7287225Z * [new tag] viable/strict/1761686693 -> viable/strict/1761686693 2025-12-04T08:57:44.7288134Z * [new tag] viable/strict/1761688179 -> viable/strict/1761688179 2025-12-04T08:57:44.7289097Z * [new tag] viable/strict/1761691973 -> viable/strict/1761691973 2025-12-04T08:57:44.7290185Z * [new tag] viable/strict/1761693884 -> viable/strict/1761693884 2025-12-04T08:57:44.7291090Z * [new tag] viable/strict/1761695389 -> viable/strict/1761695389 2025-12-04T08:57:44.7292055Z * [new tag] viable/strict/1761698408 -> viable/strict/1761698408 2025-12-04T08:57:44.7292963Z * [new tag] viable/strict/1761702931 -> viable/strict/1761702931 2025-12-04T08:57:44.7293972Z * [new tag] viable/strict/1761706307 -> viable/strict/1761706307 2025-12-04T08:57:44.7294893Z * [new tag] viable/strict/1761709065 -> viable/strict/1761709065 2025-12-04T08:57:44.7295917Z * [new tag] viable/strict/1761710285 -> viable/strict/1761710285 2025-12-04T08:57:44.7296916Z * [new tag] viable/strict/1761711983 -> viable/strict/1761711983 2025-12-04T08:57:44.7297912Z * [new tag] viable/strict/1761713514 -> viable/strict/1761713514 2025-12-04T08:57:44.7298953Z * [new tag] viable/strict/1761715523 -> viable/strict/1761715523 2025-12-04T08:57:44.7299951Z * [new tag] viable/strict/1761727973 -> viable/strict/1761727973 2025-12-04T08:57:44.7300966Z * [new tag] viable/strict/1761751558 -> viable/strict/1761751558 2025-12-04T08:57:44.7301917Z * [new tag] viable/strict/1761755187 -> viable/strict/1761755187 2025-12-04T08:57:44.7302954Z * [new tag] viable/strict/1761756826 -> viable/strict/1761756826 2025-12-04T08:57:44.7303932Z * [new tag] viable/strict/1761769551 -> viable/strict/1761769551 2025-12-04T08:57:44.7304985Z * [new tag] viable/strict/1761771032 -> viable/strict/1761771032 2025-12-04T08:57:44.7305751Z * [new tag] viable/strict/1761773101 -> viable/strict/1761773101 2025-12-04T08:57:44.7306767Z * [new tag] viable/strict/1761781792 -> viable/strict/1761781792 2025-12-04T08:57:44.7308065Z * [new tag] viable/strict/1761784788 -> viable/strict/1761784788 2025-12-04T08:57:44.7309100Z * [new tag] viable/strict/1761786740 -> viable/strict/1761786740 2025-12-04T08:57:44.7310204Z * [new tag] viable/strict/1761789332 -> viable/strict/1761789332 2025-12-04T08:57:44.7311658Z * [new tag] viable/strict/1761792569 -> viable/strict/1761792569 2025-12-04T08:57:44.7312764Z * [new tag] viable/strict/1761795289 -> viable/strict/1761795289 2025-12-04T08:57:44.7313741Z * [new tag] viable/strict/1761798345 -> viable/strict/1761798345 2025-12-04T08:57:44.7314717Z * [new tag] viable/strict/1761799827 -> viable/strict/1761799827 2025-12-04T08:57:44.7315770Z * [new tag] viable/strict/1761805604 -> viable/strict/1761805604 2025-12-04T08:57:44.7316748Z * [new tag] viable/strict/1761807202 -> viable/strict/1761807202 2025-12-04T08:57:44.7317774Z * [new tag] viable/strict/1761809094 -> viable/strict/1761809094 2025-12-04T08:57:44.7318748Z * [new tag] viable/strict/1761810576 -> viable/strict/1761810576 2025-12-04T08:57:44.7319879Z * [new tag] viable/strict/1761812771 -> viable/strict/1761812771 2025-12-04T08:57:44.7320856Z * [new tag] viable/strict/1761814363 -> viable/strict/1761814363 2025-12-04T08:57:44.7321798Z * [new tag] viable/strict/1761857410 -> viable/strict/1761857410 2025-12-04T08:57:44.7322783Z * [new tag] viable/strict/1761860985 -> viable/strict/1761860985 2025-12-04T08:57:44.7323877Z * [new tag] viable/strict/1761863094 -> viable/strict/1761863094 2025-12-04T08:57:44.7325274Z * [new tag] viable/strict/1761864590 -> viable/strict/1761864590 2025-12-04T08:57:44.7326245Z * [new tag] viable/strict/1761866675 -> viable/strict/1761866675 2025-12-04T08:57:44.7327460Z * [new tag] viable/strict/1761868178 -> viable/strict/1761868178 2025-12-04T08:57:44.7328454Z * [new tag] viable/strict/1761871111 -> viable/strict/1761871111 2025-12-04T08:57:44.7329956Z * [new tag] viable/strict/1761873126 -> viable/strict/1761873126 2025-12-04T08:57:44.7331010Z * [new tag] viable/strict/1761875714 -> viable/strict/1761875714 2025-12-04T08:57:44.7332009Z * [new tag] viable/strict/1761878924 -> viable/strict/1761878924 2025-12-04T08:57:44.7333029Z * [new tag] viable/strict/1761881727 -> viable/strict/1761881727 2025-12-04T08:57:44.7334000Z * [new tag] viable/strict/1761882959 -> viable/strict/1761882959 2025-12-04T08:57:44.7334992Z * [new tag] viable/strict/1761886268 -> viable/strict/1761886268 2025-12-04T08:57:44.7336043Z * [new tag] viable/strict/1761893641 -> viable/strict/1761893641 2025-12-04T08:57:44.7337141Z * [new tag] viable/strict/1761931517 -> viable/strict/1761931517 2025-12-04T08:57:44.7338107Z * [new tag] viable/strict/1761933080 -> viable/strict/1761933080 2025-12-04T08:57:44.7339047Z * [new tag] viable/strict/1761935217 -> viable/strict/1761935217 2025-12-04T08:57:44.7340059Z * [new tag] viable/strict/1761938533 -> viable/strict/1761938533 2025-12-04T08:57:44.7341097Z * [new tag] viable/strict/1761940184 -> viable/strict/1761940184 2025-12-04T08:57:44.7342066Z * [new tag] viable/strict/1761942338 -> viable/strict/1761942338 2025-12-04T08:57:44.7343043Z * [new tag] viable/strict/1761946100 -> viable/strict/1761946100 2025-12-04T08:57:44.7343993Z * [new tag] viable/strict/1761947374 -> viable/strict/1761947374 2025-12-04T08:57:44.7344995Z * [new tag] viable/strict/1761950978 -> viable/strict/1761950978 2025-12-04T08:57:44.7346030Z * [new tag] viable/strict/1761957727 -> viable/strict/1761957727 2025-12-04T08:57:44.7347005Z * [new tag] viable/strict/1761959532 -> viable/strict/1761959532 2025-12-04T08:57:44.7348442Z * [new tag] viable/strict/1761965366 -> viable/strict/1761965366 2025-12-04T08:57:44.7349664Z * [new tag] viable/strict/1761968066 -> viable/strict/1761968066 2025-12-04T08:57:44.7350699Z * [new tag] viable/strict/1761969322 -> viable/strict/1761969322 2025-12-04T08:57:44.7351698Z * [new tag] viable/strict/1761974723 -> viable/strict/1761974723 2025-12-04T08:57:44.7352828Z * [new tag] viable/strict/1761981837 -> viable/strict/1761981837 2025-12-04T08:57:44.7353866Z * [new tag] viable/strict/1761985546 -> viable/strict/1761985546 2025-12-04T08:57:44.7354880Z * [new tag] viable/strict/1761987030 -> viable/strict/1761987030 2025-12-04T08:57:44.7355993Z * [new tag] viable/strict/1762003554 -> viable/strict/1762003554 2025-12-04T08:57:44.7356965Z * [new tag] viable/strict/1762021560 -> viable/strict/1762021560 2025-12-04T08:57:44.7357964Z * [new tag] viable/strict/1762032190 -> viable/strict/1762032190 2025-12-04T08:57:44.7358961Z * [new tag] viable/strict/1762040981 -> viable/strict/1762040981 2025-12-04T08:57:44.7360107Z * [new tag] viable/strict/1762048525 -> viable/strict/1762048525 2025-12-04T08:57:44.7361093Z * [new tag] viable/strict/1762104223 -> viable/strict/1762104223 2025-12-04T08:57:44.7362132Z * [new tag] viable/strict/1762105778 -> viable/strict/1762105778 2025-12-04T08:57:44.7363084Z * [new tag] viable/strict/1762115109 -> viable/strict/1762115109 2025-12-04T08:57:44.7364051Z * [new tag] viable/strict/1762125840 -> viable/strict/1762125840 2025-12-04T08:57:44.7364821Z * [new tag] viable/strict/1762127377 -> viable/strict/1762127377 2025-12-04T08:57:44.7366206Z * [new tag] viable/strict/1762134925 -> viable/strict/1762134925 2025-12-04T08:57:44.7366998Z * [new tag] viable/strict/1762138338 -> viable/strict/1762138338 2025-12-04T08:57:44.7368086Z * [new tag] viable/strict/1762148993 -> viable/strict/1762148993 2025-12-04T08:57:44.7369100Z * [new tag] viable/strict/1762152871 -> viable/strict/1762152871 2025-12-04T08:57:44.7370091Z * [new tag] viable/strict/1762156183 -> viable/strict/1762156183 2025-12-04T08:57:44.7371036Z * [new tag] viable/strict/1762163457 -> viable/strict/1762163457 2025-12-04T08:57:44.7372000Z * [new tag] viable/strict/1762165569 -> viable/strict/1762165569 2025-12-04T08:57:44.7372943Z * [new tag] viable/strict/1762169035 -> viable/strict/1762169035 2025-12-04T08:57:44.7373939Z * [new tag] viable/strict/1762174936 -> viable/strict/1762174936 2025-12-04T08:57:44.7374933Z * [new tag] viable/strict/1762194412 -> viable/strict/1762194412 2025-12-04T08:57:44.7375907Z * [new tag] viable/strict/1762195876 -> viable/strict/1762195876 2025-12-04T08:57:44.7376880Z * [new tag] viable/strict/1762197788 -> viable/strict/1762197788 2025-12-04T08:57:44.7377883Z * [new tag] viable/strict/1762199389 -> viable/strict/1762199389 2025-12-04T08:57:44.7379193Z * [new tag] viable/strict/1762206585 -> viable/strict/1762206585 2025-12-04T08:57:44.7380266Z * [new tag] viable/strict/1762210184 -> viable/strict/1762210184 2025-12-04T08:57:44.7381041Z * [new tag] viable/strict/1762218736 -> viable/strict/1762218736 2025-12-04T08:57:44.7382081Z * [new tag] viable/strict/1762224529 -> viable/strict/1762224529 2025-12-04T08:57:44.7383096Z * [new tag] viable/strict/1762227253 -> viable/strict/1762227253 2025-12-04T08:57:44.7383891Z * [new tag] viable/strict/1762228515 -> viable/strict/1762228515 2025-12-04T08:57:44.7385133Z * [new tag] viable/strict/1762230349 -> viable/strict/1762230349 2025-12-04T08:57:44.7385989Z * [new tag] viable/strict/1762231859 -> viable/strict/1762231859 2025-12-04T08:57:44.7387006Z * [new tag] viable/strict/1762233925 -> viable/strict/1762233925 2025-12-04T08:57:44.7388558Z * [new tag] viable/strict/1762237630 -> viable/strict/1762237630 2025-12-04T08:57:44.7389398Z * [new tag] viable/strict/1762253522 -> viable/strict/1762253522 2025-12-04T08:57:44.7390546Z * [new tag] viable/strict/1762278588 -> viable/strict/1762278588 2025-12-04T08:57:44.7391579Z * [new tag] viable/strict/1762284203 -> viable/strict/1762284203 2025-12-04T08:57:44.7392611Z * [new tag] viable/strict/1762289446 -> viable/strict/1762289446 2025-12-04T08:57:44.7393641Z * [new tag] viable/strict/1762291515 -> viable/strict/1762291515 2025-12-04T08:57:44.7394666Z * [new tag] viable/strict/1762295100 -> viable/strict/1762295100 2025-12-04T08:57:44.7395985Z * [new tag] viable/strict/1762296590 -> viable/strict/1762296590 2025-12-04T08:57:44.7396805Z * [new tag] viable/strict/1762300179 -> viable/strict/1762300179 2025-12-04T08:57:44.7397649Z * [new tag] viable/strict/1762303207 -> viable/strict/1762303207 2025-12-04T08:57:44.7398754Z * [new tag] viable/strict/1762386584 -> viable/strict/1762386584 2025-12-04T08:57:44.7399844Z * [new tag] viable/strict/1762391537 -> viable/strict/1762391537 2025-12-04T08:57:44.7400655Z * [new tag] viable/strict/1762394119 -> viable/strict/1762394119 2025-12-04T08:57:44.7402061Z * [new tag] viable/strict/1762397437 -> viable/strict/1762397437 2025-12-04T08:57:44.7403075Z * [new tag] viable/strict/1762400256 -> viable/strict/1762400256 2025-12-04T08:57:44.7404076Z * [new tag] viable/strict/1762401469 -> viable/strict/1762401469 2025-12-04T08:57:44.7405080Z * [new tag] viable/strict/1762408195 -> viable/strict/1762408195 2025-12-04T08:57:44.7406095Z * [new tag] viable/strict/1762410411 -> viable/strict/1762410411 2025-12-04T08:57:44.7407101Z * [new tag] viable/strict/1762417613 -> viable/strict/1762417613 2025-12-04T08:57:44.7408120Z * [new tag] viable/strict/1762419198 -> viable/strict/1762419198 2025-12-04T08:57:44.7409083Z * [new tag] viable/strict/1762422656 -> viable/strict/1762422656 2025-12-04T08:57:44.7410520Z * [new tag] viable/strict/1762424746 -> viable/strict/1762424746 2025-12-04T08:57:44.7411598Z * [new tag] viable/strict/1762446386 -> viable/strict/1762446386 2025-12-04T08:57:44.7412674Z * [new tag] viable/strict/1762449912 -> viable/strict/1762449912 2025-12-04T08:57:44.7413692Z * [new tag] viable/strict/1762457031 -> viable/strict/1762457031 2025-12-04T08:57:44.7414672Z * [new tag] viable/strict/1762462441 -> viable/strict/1762462441 2025-12-04T08:57:44.7434666Z * [new tag] viable/strict/1762467909 -> viable/strict/1762467909 2025-12-04T08:57:44.7435012Z * [new tag] viable/strict/1762471493 -> viable/strict/1762471493 2025-12-04T08:57:44.7435237Z * [new tag] viable/strict/1762475990 -> viable/strict/1762475990 2025-12-04T08:57:44.7435451Z * [new tag] viable/strict/1762477933 -> viable/strict/1762477933 2025-12-04T08:57:44.7435678Z * [new tag] viable/strict/1762491053 -> viable/strict/1762491053 2025-12-04T08:57:44.7435894Z * [new tag] viable/strict/1762493118 -> viable/strict/1762493118 2025-12-04T08:57:44.7436115Z * [new tag] viable/strict/1762498442 -> viable/strict/1762498442 2025-12-04T08:57:44.7436488Z * [new tag] viable/strict/1762501778 -> viable/strict/1762501778 2025-12-04T08:57:44.7436811Z * [new tag] viable/strict/1762504001 -> viable/strict/1762504001 2025-12-04T08:57:44.7437036Z * [new tag] viable/strict/1762505583 -> viable/strict/1762505583 2025-12-04T08:57:44.7437248Z * [new tag] viable/strict/1762507523 -> viable/strict/1762507523 2025-12-04T08:57:44.7437471Z * [new tag] viable/strict/1762511140 -> viable/strict/1762511140 2025-12-04T08:57:44.7437683Z * [new tag] viable/strict/1762512632 -> viable/strict/1762512632 2025-12-04T08:57:44.7437892Z * [new tag] viable/strict/1762520467 -> viable/strict/1762520467 2025-12-04T08:57:44.7438115Z * [new tag] viable/strict/1762522016 -> viable/strict/1762522016 2025-12-04T08:57:44.7438331Z * [new tag] viable/strict/1762530591 -> viable/strict/1762530591 2025-12-04T08:57:44.7438545Z * [new tag] viable/strict/1762543405 -> viable/strict/1762543405 2025-12-04T08:57:44.7438773Z * [new tag] viable/strict/1762544998 -> viable/strict/1762544998 2025-12-04T08:57:44.7439091Z * [new tag] viable/strict/1762552182 -> viable/strict/1762552182 2025-12-04T08:57:44.7439305Z * [new tag] viable/strict/1762554297 -> viable/strict/1762554297 2025-12-04T08:57:44.7439509Z * [new tag] viable/strict/1762559381 -> viable/strict/1762559381 2025-12-04T08:57:44.7439708Z * [new tag] viable/strict/1762562222 -> viable/strict/1762562222 2025-12-04T08:57:44.7439974Z * [new tag] viable/strict/1762564319 -> viable/strict/1762564319 2025-12-04T08:57:44.7440181Z * [new tag] viable/strict/1762566904 -> viable/strict/1762566904 2025-12-04T08:57:44.7440762Z * [new tag] viable/strict/1762569781 -> viable/strict/1762569781 2025-12-04T08:57:44.7441757Z * [new tag] viable/strict/1762575940 -> viable/strict/1762575940 2025-12-04T08:57:44.7442752Z * [new tag] viable/strict/1762580974 -> viable/strict/1762580974 2025-12-04T08:57:44.7443722Z * [new tag] viable/strict/1762583185 -> viable/strict/1762583185 2025-12-04T08:57:44.7444715Z * [new tag] viable/strict/1762586647 -> viable/strict/1762586647 2025-12-04T08:57:44.7445844Z * [new tag] viable/strict/1762588183 -> viable/strict/1762588183 2025-12-04T08:57:44.7446837Z * [new tag] viable/strict/1762593886 -> viable/strict/1762593886 2025-12-04T08:57:44.7447836Z * [new tag] viable/strict/1762650743 -> viable/strict/1762650743 2025-12-04T08:57:44.7448861Z * [new tag] viable/strict/1762653328 -> viable/strict/1762653328 2025-12-04T08:57:44.7449876Z * [new tag] viable/strict/1762659342 -> viable/strict/1762659342 2025-12-04T08:57:44.7450854Z * [new tag] viable/strict/1762662360 -> viable/strict/1762662360 2025-12-04T08:57:44.7451811Z * [new tag] viable/strict/1762667377 -> viable/strict/1762667377 2025-12-04T08:57:44.7453070Z * [new tag] viable/strict/1762671090 -> viable/strict/1762671090 2025-12-04T08:57:44.7454043Z * [new tag] viable/strict/1762680284 -> viable/strict/1762680284 2025-12-04T08:57:44.7455047Z * [new tag] viable/strict/1762683900 -> viable/strict/1762683900 2025-12-04T08:57:44.7456039Z * [new tag] viable/strict/1762705541 -> viable/strict/1762705541 2025-12-04T08:57:44.7457076Z * [new tag] viable/strict/1762709004 -> viable/strict/1762709004 2025-12-04T08:57:44.7458068Z * [new tag] viable/strict/1762746004 -> viable/strict/1762746004 2025-12-04T08:57:44.7459169Z * [new tag] viable/strict/1762748799 -> viable/strict/1762748799 2025-12-04T08:57:44.7460292Z * [new tag] viable/strict/1762759504 -> viable/strict/1762759504 2025-12-04T08:57:44.7461340Z * [new tag] viable/strict/1762760973 -> viable/strict/1762760973 2025-12-04T08:57:44.7463032Z * [new tag] viable/strict/1762775374 -> viable/strict/1762775374 2025-12-04T08:57:44.7464063Z * [new tag] viable/strict/1762777661 -> viable/strict/1762777661 2025-12-04T08:57:44.7465037Z * [new tag] viable/strict/1762779774 -> viable/strict/1762779774 2025-12-04T08:57:44.7466228Z * [new tag] viable/strict/1762781259 -> viable/strict/1762781259 2025-12-04T08:57:44.7468045Z * [new tag] viable/strict/1762793628 -> viable/strict/1762793628 2025-12-04T08:57:44.7469815Z * [new tag] viable/strict/1762800711 -> viable/strict/1762800711 2025-12-04T08:57:44.7470028Z * [new tag] viable/strict/1762809894 -> viable/strict/1762809894 2025-12-04T08:57:44.7470758Z * [new tag] viable/strict/1762811384 -> viable/strict/1762811384 2025-12-04T08:57:44.7471817Z * [new tag] viable/strict/1762813841 -> viable/strict/1762813841 2025-12-04T08:57:44.7472835Z * [new tag] viable/strict/1762815047 -> viable/strict/1762815047 2025-12-04T08:57:44.7474104Z * [new tag] viable/strict/1762817094 -> viable/strict/1762817094 2025-12-04T08:57:44.7475135Z * [new tag] viable/strict/1762818582 -> viable/strict/1762818582 2025-12-04T08:57:44.7476157Z * [new tag] viable/strict/1762821623 -> viable/strict/1762821623 2025-12-04T08:57:44.7476966Z * [new tag] viable/strict/1762823531 -> viable/strict/1762823531 2025-12-04T08:57:44.7478098Z * [new tag] viable/strict/1762849583 -> viable/strict/1762849583 2025-12-04T08:57:44.7479145Z * [new tag] viable/strict/1762851200 -> viable/strict/1762851200 2025-12-04T08:57:44.7480365Z * [new tag] viable/strict/1762854603 -> viable/strict/1762854603 2025-12-04T08:57:44.7481457Z * [new tag] viable/strict/1762858276 -> viable/strict/1762858276 2025-12-04T08:57:44.7482472Z * [new tag] viable/strict/1762860891 -> viable/strict/1762860891 2025-12-04T08:57:44.7484076Z * [new tag] viable/strict/1762866174 -> viable/strict/1762866174 2025-12-04T08:57:44.7485076Z * [new tag] viable/strict/1762867653 -> viable/strict/1762867653 2025-12-04T08:57:44.7486050Z * [new tag] viable/strict/1762872669 -> viable/strict/1762872669 2025-12-04T08:57:44.7486841Z * [new tag] viable/strict/1762878380 -> viable/strict/1762878380 2025-12-04T08:57:44.7487910Z * [new tag] viable/strict/1762889003 -> viable/strict/1762889003 2025-12-04T08:57:44.7488951Z * [new tag] viable/strict/1762890589 -> viable/strict/1762890589 2025-12-04T08:57:44.7489947Z * [new tag] viable/strict/1762892743 -> viable/strict/1762892743 2025-12-04T08:57:44.7490956Z * [new tag] viable/strict/1762894271 -> viable/strict/1762894271 2025-12-04T08:57:44.7491749Z * [new tag] viable/strict/1762896287 -> viable/strict/1762896287 2025-12-04T08:57:44.7492765Z * [new tag] viable/strict/1762915871 -> viable/strict/1762915871 2025-12-04T08:57:44.7493814Z * [new tag] viable/strict/1762918569 -> viable/strict/1762918569 2025-12-04T08:57:44.7494591Z * [new tag] viable/strict/1762919776 -> viable/strict/1762919776 2025-12-04T08:57:44.7495670Z * [new tag] viable/strict/1762923072 -> viable/strict/1762923072 2025-12-04T08:57:44.7496662Z * [new tag] viable/strict/1762928826 -> viable/strict/1762928826 2025-12-04T08:57:44.7497768Z * [new tag] viable/strict/1762930451 -> viable/strict/1762930451 2025-12-04T08:57:44.7498803Z * [new tag] viable/strict/1762933780 -> viable/strict/1762933780 2025-12-04T08:57:44.7499786Z * [new tag] viable/strict/1762937638 -> viable/strict/1762937638 2025-12-04T08:57:44.7500987Z * [new tag] viable/strict/1762939545 -> viable/strict/1762939545 2025-12-04T08:57:44.7502004Z * [new tag] viable/strict/1762962692 -> viable/strict/1762962692 2025-12-04T08:57:44.7503028Z * [new tag] viable/strict/1762979143 -> viable/strict/1762979143 2025-12-04T08:57:44.7504009Z * [new tag] viable/strict/1762984188 -> viable/strict/1762984188 2025-12-04T08:57:44.7504796Z * [new tag] viable/strict/1762986306 -> viable/strict/1762986306 2025-12-04T08:57:44.7505843Z * [new tag] viable/strict/1762989903 -> viable/strict/1762989903 2025-12-04T08:57:44.7506813Z * [new tag] viable/strict/1762991377 -> viable/strict/1762991377 2025-12-04T08:57:44.7508149Z * [new tag] viable/strict/1762998921 -> viable/strict/1762998921 2025-12-04T08:57:44.7509343Z * [new tag] viable/strict/1763002287 -> viable/strict/1763002287 2025-12-04T08:57:44.7510419Z * [new tag] viable/strict/1763016840 -> viable/strict/1763016840 2025-12-04T08:57:44.7511416Z * [new tag] viable/strict/1763020180 -> viable/strict/1763020180 2025-12-04T08:57:44.7512517Z * [new tag] viable/strict/1763027421 -> viable/strict/1763027421 2025-12-04T08:57:44.7513646Z * [new tag] viable/strict/1763031120 -> viable/strict/1763031120 2025-12-04T08:57:44.7514710Z * [new tag] viable/strict/1763036861 -> viable/strict/1763036861 2025-12-04T08:57:44.7515747Z * [new tag] viable/strict/1763038993 -> viable/strict/1763038993 2025-12-04T08:57:44.7516842Z * [new tag] viable/strict/1763054703 -> viable/strict/1763054703 2025-12-04T08:57:44.7517697Z * [new tag] viable/strict/1763067061 -> viable/strict/1763067061 2025-12-04T08:57:44.7518775Z * [new tag] viable/strict/1763070847 -> viable/strict/1763070847 2025-12-04T08:57:44.7519957Z * [new tag] viable/strict/1763072706 -> viable/strict/1763072706 2025-12-04T08:57:44.7521062Z * [new tag] viable/strict/1763076302 -> viable/strict/1763076302 2025-12-04T08:57:44.7522025Z * [new tag] viable/strict/1763080816 -> viable/strict/1763080816 2025-12-04T08:57:44.7523040Z * [new tag] viable/strict/1763082732 -> viable/strict/1763082732 2025-12-04T08:57:44.7524428Z * [new tag] viable/strict/1763085329 -> viable/strict/1763085329 2025-12-04T08:57:44.7525545Z * [new tag] viable/strict/1763088623 -> viable/strict/1763088623 2025-12-04T08:57:44.7526661Z * [new tag] viable/strict/1763091402 -> viable/strict/1763091402 2025-12-04T08:57:44.7527693Z * [new tag] viable/strict/1763092602 -> viable/strict/1763092602 2025-12-04T08:57:44.7528736Z * [new tag] viable/strict/1763094355 -> viable/strict/1763094355 2025-12-04T08:57:44.7529813Z * [new tag] viable/strict/1763099390 -> viable/strict/1763099390 2025-12-04T08:57:44.7531261Z * [new tag] viable/strict/1763101608 -> viable/strict/1763101608 2025-12-04T08:57:44.7532311Z * [new tag] viable/strict/1763105102 -> viable/strict/1763105102 2025-12-04T08:57:44.7533397Z * [new tag] viable/strict/1763112347 -> viable/strict/1763112347 2025-12-04T08:57:44.7534413Z * [new tag] viable/strict/1763119471 -> viable/strict/1763119471 2025-12-04T08:57:44.7535240Z * [new tag] viable/strict/1763126835 -> viable/strict/1763126835 2025-12-04T08:57:44.7536295Z * [new tag] viable/strict/1763149779 -> viable/strict/1763149779 2025-12-04T08:57:44.7537466Z * [new tag] viable/strict/1763164178 -> viable/strict/1763164178 2025-12-04T08:57:44.7538272Z * [new tag] viable/strict/1763167104 -> viable/strict/1763167104 2025-12-04T08:57:44.7539282Z * [new tag] viable/strict/1763169132 -> viable/strict/1763169132 2025-12-04T08:57:44.7540259Z * [new tag] viable/strict/1763171708 -> viable/strict/1763171708 2025-12-04T08:57:44.7541253Z * [new tag] viable/strict/1763174759 -> viable/strict/1763174759 2025-12-04T08:57:44.7542256Z * [new tag] viable/strict/1763180744 -> viable/strict/1763180744 2025-12-04T08:57:44.7543276Z * [new tag] viable/strict/1763182227 -> viable/strict/1763182227 2025-12-04T08:57:44.7544232Z * [new tag] viable/strict/1763184309 -> viable/strict/1763184309 2025-12-04T08:57:44.7545765Z * [new tag] viable/strict/1763187991 -> viable/strict/1763187991 2025-12-04T08:57:44.7546798Z * [new tag] viable/strict/1763191445 -> viable/strict/1763191445 2025-12-04T08:57:44.7548532Z * [new tag] viable/strict/1763195152 -> viable/strict/1763195152 2025-12-04T08:57:44.7549365Z * [new tag] viable/strict/1763205769 -> viable/strict/1763205769 2025-12-04T08:57:44.7550475Z * [new tag] viable/strict/1763246990 -> viable/strict/1763246990 2025-12-04T08:57:44.7551571Z * [new tag] viable/strict/1763261578 -> viable/strict/1763261578 2025-12-04T08:57:44.7552415Z * [new tag] viable/strict/1763286573 -> viable/strict/1763286573 2025-12-04T08:57:44.7553359Z * [new tag] viable/strict/1763292167 -> viable/strict/1763292167 2025-12-04T08:57:44.7554441Z * [new tag] viable/strict/1763333386 -> viable/strict/1763333386 2025-12-04T08:57:44.7555421Z * [new tag] viable/strict/1763340082 -> viable/strict/1763340082 2025-12-04T08:57:44.7557258Z * [new tag] viable/strict/1763364324 -> viable/strict/1763364324 2025-12-04T08:57:44.7558290Z * [new tag] viable/strict/1763371569 -> viable/strict/1763371569 2025-12-04T08:57:44.7559362Z * [new tag] viable/strict/1763373067 -> viable/strict/1763373067 2025-12-04T08:57:44.7560474Z * [new tag] viable/strict/1763375157 -> viable/strict/1763375157 2025-12-04T08:57:44.7561654Z * [new tag] viable/strict/1763382462 -> viable/strict/1763382462 2025-12-04T08:57:44.7562683Z * [new tag] viable/strict/1763394661 -> viable/strict/1763394661 2025-12-04T08:57:44.7563942Z * [new tag] viable/strict/1763396797 -> viable/strict/1763396797 2025-12-04T08:57:44.7564986Z * [new tag] viable/strict/1763398542 -> viable/strict/1763398542 2025-12-04T08:57:44.7566028Z * [new tag] viable/strict/1763401807 -> viable/strict/1763401807 2025-12-04T08:57:44.7566966Z * [new tag] viable/strict/1763414698 -> viable/strict/1763414698 2025-12-04T08:57:44.7567918Z * [new tag] viable/strict/1763419807 -> viable/strict/1763419807 2025-12-04T08:57:44.7568912Z * [new tag] viable/strict/1763426369 -> viable/strict/1763426369 2025-12-04T08:57:44.7570069Z * [new tag] viable/strict/1763428331 -> viable/strict/1763428331 2025-12-04T08:57:44.7571130Z * [new tag] viable/strict/1763430922 -> viable/strict/1763430922 2025-12-04T08:57:44.7571941Z * [new tag] viable/strict/1763434184 -> viable/strict/1763434184 2025-12-04T08:57:44.7572984Z * [new tag] viable/strict/1763439973 -> viable/strict/1763439973 2025-12-04T08:57:44.7574047Z * [new tag] viable/strict/1763444995 -> viable/strict/1763444995 2025-12-04T08:57:44.7575095Z * [new tag] viable/strict/1763447206 -> viable/strict/1763447206 2025-12-04T08:57:44.7576123Z * [new tag] viable/strict/1763448826 -> viable/strict/1763448826 2025-12-04T08:57:44.7577070Z * [new tag] viable/strict/1763450717 -> viable/strict/1763450717 2025-12-04T08:57:44.7578117Z * [new tag] viable/strict/1763452183 -> viable/strict/1763452183 2025-12-04T08:57:44.7579190Z * [new tag] viable/strict/1763457945 -> viable/strict/1763457945 2025-12-04T08:57:44.7580167Z * [new tag] viable/strict/1763459439 -> viable/strict/1763459439 2025-12-04T08:57:44.7581078Z * [new tag] viable/strict/1763461556 -> viable/strict/1763461556 2025-12-04T08:57:44.7582151Z * [new tag] viable/strict/1763463103 -> viable/strict/1763463103 2025-12-04T08:57:44.7583242Z * [new tag] viable/strict/1763465100 -> viable/strict/1763465100 2025-12-04T08:57:44.7584061Z * [new tag] viable/strict/1763468866 -> viable/strict/1763468866 2025-12-04T08:57:44.7584954Z * [new tag] viable/strict/1763493823 -> viable/strict/1763493823 2025-12-04T08:57:44.7585770Z * [new tag] viable/strict/1763496249 -> viable/strict/1763496249 2025-12-04T08:57:44.7586810Z * [new tag] viable/strict/1763502620 -> viable/strict/1763502620 2025-12-04T08:57:44.7588247Z * [new tag] viable/strict/1763504715 -> viable/strict/1763504715 2025-12-04T08:57:44.7589237Z * [new tag] viable/strict/1763506208 -> viable/strict/1763506208 2025-12-04T08:57:44.7590272Z * [new tag] viable/strict/1763520590 -> viable/strict/1763520590 2025-12-04T08:57:44.7591357Z * [new tag] viable/strict/1763523357 -> viable/strict/1763523357 2025-12-04T08:57:44.7592432Z * [new tag] viable/strict/1763529922 -> viable/strict/1763529922 2025-12-04T08:57:44.7593543Z * [new tag] viable/strict/1763531408 -> viable/strict/1763531408 2025-12-04T08:57:44.7594563Z * [new tag] viable/strict/1763533622 -> viable/strict/1763533622 2025-12-04T08:57:44.7595577Z * [new tag] viable/strict/1763538576 -> viable/strict/1763538576 2025-12-04T08:57:44.7596706Z * [new tag] viable/strict/1763545823 -> viable/strict/1763545823 2025-12-04T08:57:44.7597488Z * [new tag] viable/strict/1763547951 -> viable/strict/1763547951 2025-12-04T08:57:44.7599105Z * [new tag] viable/strict/1763551477 -> viable/strict/1763551477 2025-12-04T08:57:44.7600208Z * [new tag] viable/strict/1763552982 -> viable/strict/1763552982 2025-12-04T08:57:44.7601185Z * [new tag] viable/strict/1763594698 -> viable/strict/1763594698 2025-12-04T08:57:44.7602201Z * [new tag] viable/strict/1763596178 -> viable/strict/1763596178 2025-12-04T08:57:44.7603214Z * [new tag] viable/strict/1763599155 -> viable/strict/1763599155 2025-12-04T08:57:44.7604283Z * [new tag] viable/strict/1763603717 -> viable/strict/1763603717 2025-12-04T08:57:44.7605284Z * [new tag] viable/strict/1763606923 -> viable/strict/1763606923 2025-12-04T08:57:44.7606295Z * [new tag] viable/strict/1763609715 -> viable/strict/1763609715 2025-12-04T08:57:44.7607251Z * [new tag] viable/strict/1763612757 -> viable/strict/1763612757 2025-12-04T08:57:44.7608262Z * [new tag] viable/strict/1763616325 -> viable/strict/1763616325 2025-12-04T08:57:44.7609263Z * [new tag] viable/strict/1763623509 -> viable/strict/1763623509 2025-12-04T08:57:44.7610380Z * [new tag] viable/strict/1763624984 -> viable/strict/1763624984 2025-12-04T08:57:44.7611376Z * [new tag] viable/strict/1763628796 -> viable/strict/1763628796 2025-12-04T08:57:44.7612425Z * [new tag] viable/strict/1763634343 -> viable/strict/1763634343 2025-12-04T08:57:44.7613243Z * [new tag] viable/strict/1763635867 -> viable/strict/1763635867 2025-12-04T08:57:44.7614485Z * [new tag] viable/strict/1763639382 -> viable/strict/1763639382 2025-12-04T08:57:44.7615568Z * [new tag] viable/strict/1763646626 -> viable/strict/1763646626 2025-12-04T08:57:44.7616738Z * [new tag] viable/strict/1763655997 -> viable/strict/1763655997 2025-12-04T08:57:44.7617702Z * [new tag] viable/strict/1763659444 -> viable/strict/1763659444 2025-12-04T08:57:44.7618687Z * [new tag] viable/strict/1763660992 -> viable/strict/1763660992 2025-12-04T08:57:44.7619637Z * [new tag] viable/strict/1763663201 -> viable/strict/1763663201 2025-12-04T08:57:44.7620753Z * [new tag] viable/strict/1763670362 -> viable/strict/1763670362 2025-12-04T08:57:44.7621510Z * [new tag] viable/strict/1763675378 -> viable/strict/1763675378 2025-12-04T08:57:44.7622550Z * [new tag] viable/strict/1763693343 -> viable/strict/1763693343 2025-12-04T08:57:44.7623684Z * [new tag] viable/strict/1763696088 -> viable/strict/1763696088 2025-12-04T08:57:44.7625242Z * [new tag] viable/strict/1763697343 -> viable/strict/1763697343 2025-12-04T08:57:44.7626277Z * [new tag] viable/strict/1763699165 -> viable/strict/1763699165 2025-12-04T08:57:44.7627343Z * [new tag] viable/strict/1763700660 -> viable/strict/1763700660 2025-12-04T08:57:44.7628415Z * [new tag] viable/strict/1763704209 -> viable/strict/1763704209 2025-12-04T08:57:44.7629468Z * [new tag] viable/strict/1763706411 -> viable/strict/1763706411 2025-12-04T08:57:44.7630421Z * [new tag] viable/strict/1763708082 -> viable/strict/1763708082 2025-12-04T08:57:44.7631376Z * [new tag] viable/strict/1763711381 -> viable/strict/1763711381 2025-12-04T08:57:44.7632220Z * [new tag] viable/strict/1763713593 -> viable/strict/1763713593 2025-12-04T08:57:44.7633351Z * [new tag] viable/strict/1763715201 -> viable/strict/1763715201 2025-12-04T08:57:44.7634337Z * [new tag] viable/strict/1763733017 -> viable/strict/1763733017 2025-12-04T08:57:44.7635414Z * [new tag] viable/strict/1763735108 -> viable/strict/1763735108 2025-12-04T08:57:44.7636415Z * [new tag] viable/strict/1763749579 -> viable/strict/1763749579 2025-12-04T08:57:44.7637435Z * [new tag] viable/strict/1763751113 -> viable/strict/1763751113 2025-12-04T08:57:44.7638477Z * [new tag] viable/strict/1763753035 -> viable/strict/1763753035 2025-12-04T08:57:44.7639583Z * [new tag] viable/strict/1763754578 -> viable/strict/1763754578 2025-12-04T08:57:44.7640662Z * [new tag] viable/strict/1763756748 -> viable/strict/1763756748 2025-12-04T08:57:44.7641605Z * [new tag] viable/strict/1763758205 -> viable/strict/1763758205 2025-12-04T08:57:44.7642392Z * [new tag] viable/strict/1763764050 -> viable/strict/1763764050 2025-12-04T08:57:44.7643400Z * [new tag] viable/strict/1763771887 -> viable/strict/1763771887 2025-12-04T08:57:44.7644545Z * [new tag] viable/strict/1763773920 -> viable/strict/1763773920 2025-12-04T08:57:44.7645491Z * [new tag] viable/strict/1763776501 -> viable/strict/1763776501 2025-12-04T08:57:44.7646458Z * [new tag] viable/strict/1763779437 -> viable/strict/1763779437 2025-12-04T08:57:44.7647686Z * [new tag] viable/strict/1763781038 -> viable/strict/1763781038 2025-12-04T08:57:44.7648608Z * [new tag] viable/strict/1763782245 -> viable/strict/1763782245 2025-12-04T08:57:44.7649762Z * [new tag] viable/strict/1763785568 -> viable/strict/1763785568 2025-12-04T08:57:44.7650676Z * [new tag] viable/strict/1763787006 -> viable/strict/1763787006 2025-12-04T08:57:44.7651722Z * [new tag] viable/strict/1763789103 -> viable/strict/1763789103 2025-12-04T08:57:44.7652703Z * [new tag] viable/strict/1763790578 -> viable/strict/1763790578 2025-12-04T08:57:44.7653664Z * [new tag] viable/strict/1763796275 -> viable/strict/1763796275 2025-12-04T08:57:44.7654943Z * [new tag] viable/strict/1763801465 -> viable/strict/1763801465 2025-12-04T08:57:44.7655909Z * [new tag] viable/strict/1763803522 -> viable/strict/1763803522 2025-12-04T08:57:44.7656907Z * [new tag] viable/strict/1763808581 -> viable/strict/1763808581 2025-12-04T08:57:44.7657882Z * [new tag] viable/strict/1763840977 -> viable/strict/1763840977 2025-12-04T08:57:44.7658840Z * [new tag] viable/strict/1763846659 -> viable/strict/1763846659 2025-12-04T08:57:44.7659802Z * [new tag] viable/strict/1763872065 -> viable/strict/1763872065 2025-12-04T08:57:44.7660835Z * [new tag] viable/strict/1763873648 -> viable/strict/1763873648 2025-12-04T08:57:44.7661859Z * [new tag] viable/strict/1763875506 -> viable/strict/1763875506 2025-12-04T08:57:44.7662625Z * [new tag] viable/strict/1763889904 -> viable/strict/1763889904 2025-12-04T08:57:44.7663647Z * [new tag] viable/strict/1763930999 -> viable/strict/1763930999 2025-12-04T08:57:44.7665153Z * [new tag] viable/strict/1763944964 -> viable/strict/1763944964 2025-12-04T08:57:44.7665940Z * [new tag] viable/strict/1763958474 -> viable/strict/1763958474 2025-12-04T08:57:44.7667021Z * [new tag] viable/strict/1763967263 -> viable/strict/1763967263 2025-12-04T08:57:44.7668377Z * [new tag] viable/strict/1763972803 -> viable/strict/1763972803 2025-12-04T08:57:44.7669392Z * [new tag] viable/strict/1763976376 -> viable/strict/1763976376 2025-12-04T08:57:44.7670489Z * [new tag] viable/strict/1763989404 -> viable/strict/1763989404 2025-12-04T08:57:44.7671450Z * [new tag] viable/strict/1763990887 -> viable/strict/1763990887 2025-12-04T08:57:44.7672460Z * [new tag] viable/strict/1764019919 -> viable/strict/1764019919 2025-12-04T08:57:44.7673521Z * [new tag] viable/strict/1764023134 -> viable/strict/1764023134 2025-12-04T08:57:44.7674324Z * [new tag] viable/strict/1764024593 -> viable/strict/1764024593 2025-12-04T08:57:44.7675391Z * [new tag] viable/strict/1764026706 -> viable/strict/1764026706 2025-12-04T08:57:44.7676703Z * [new tag] viable/strict/1764031139 -> viable/strict/1764031139 2025-12-04T08:57:44.7677698Z * [new tag] viable/strict/1764033131 -> viable/strict/1764033131 2025-12-04T08:57:44.7678513Z * [new tag] viable/strict/1764035725 -> viable/strict/1764035725 2025-12-04T08:57:44.7679444Z * [new tag] viable/strict/1764624265 -> viable/strict/1764624265 2025-12-04T08:57:44.7680326Z * [new tag] viable/strict/1764631514 -> viable/strict/1764631514 2025-12-04T08:57:44.7681314Z * [new tag] viable/strict/1764632987 -> viable/strict/1764632987 2025-12-04T08:57:44.7682060Z * [new tag] viable/strict/1764636063 -> viable/strict/1764636063 2025-12-04T08:57:44.7682879Z * [new tag] viable/strict/1764643975 -> viable/strict/1764643975 2025-12-04T08:57:44.7683673Z * [new tag] viable/strict/1764646859 -> viable/strict/1764646859 2025-12-04T08:57:44.7684583Z * [new tag] viable/strict/1764653120 -> viable/strict/1764653120 2025-12-04T08:57:44.7685365Z * [new tag] viable/strict/1764654632 -> viable/strict/1764654632 2025-12-04T08:57:44.7686097Z * [new tag] viable/strict/1764656821 -> viable/strict/1764656821 2025-12-04T08:57:44.7686918Z * [new tag] viable/strict/1764658557 -> viable/strict/1764658557 2025-12-04T08:57:44.7687826Z * [new tag] viable/strict/1764660333 -> viable/strict/1764660333 2025-12-04T08:57:44.7688554Z * [new tag] viable/strict/1764661812 -> viable/strict/1764661812 2025-12-04T08:57:44.7689371Z * [new tag] viable/strict/1764664023 -> viable/strict/1764664023 2025-12-04T08:57:44.7690183Z * [new tag] viable/strict/1764669150 -> viable/strict/1764669150 2025-12-04T08:57:44.7691087Z * [new tag] viable/strict/1764680709 -> viable/strict/1764680709 2025-12-04T08:57:44.7691803Z * [new tag] viable/strict/1764687619 -> viable/strict/1764687619 2025-12-04T08:57:44.7692633Z * [new tag] viable/strict/1764696355 -> viable/strict/1764696355 2025-12-04T08:57:44.7693528Z * [new tag] viable/strict/1764701767 -> viable/strict/1764701767 2025-12-04T08:57:44.7694295Z * [new tag] viable/strict/1764710768 -> viable/strict/1764710768 2025-12-04T08:57:44.7695114Z * [new tag] viable/strict/1764716202 -> viable/strict/1764716202 2025-12-04T08:57:44.7695916Z * [new tag] viable/strict/1764793566 -> viable/strict/1764793566 2025-12-04T08:57:44.7696864Z * [new tag] viable/strict/1764797093 -> viable/strict/1764797093 2025-12-04T08:57:44.7697556Z * [new tag] viable/strict/1764800729 -> viable/strict/1764800729 2025-12-04T08:57:44.7698601Z * [new tag] whc_flight_1 -> whc_flight_1 2025-12-04T08:57:44.7699557Z * [new tag] whc_flight_2 -> whc_flight_2 2025-12-04T08:57:44.7700748Z * [new tag] whc_flight_4 -> whc_flight_4 2025-12-04T08:57:44.8391573Z [command]/usr/bin/git rev-parse --verify --quiet ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32^{object} 2025-12-04T08:57:44.8416450Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:57:44.8418985Z ##[endgroup] 2025-12-04T08:57:44.8419330Z ##[group]Determining the checkout info 2025-12-04T08:57:44.8419964Z ##[endgroup] 2025-12-04T08:57:44.8424221Z [command]/usr/bin/git sparse-checkout disable 2025-12-04T08:57:44.8457625Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-12-04T08:57:44.8484790Z ##[group]Checking out the ref 2025-12-04T08:57:44.8487398Z [command]/usr/bin/git checkout --progress --force ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:57:45.8978486Z Updating files: 80% (16124/20121) 2025-12-04T08:57:45.9268416Z Updating files: 81% (16299/20121) 2025-12-04T08:57:45.9487204Z Updating files: 82% (16500/20121) 2025-12-04T08:57:45.9635920Z Updating files: 83% (16701/20121) 2025-12-04T08:57:45.9777063Z Updating files: 84% (16902/20121) 2025-12-04T08:57:45.9939551Z Updating files: 85% (17103/20121) 2025-12-04T08:57:46.0095899Z Updating files: 86% (17305/20121) 2025-12-04T08:57:46.0241414Z Updating files: 87% (17506/20121) 2025-12-04T08:57:46.0350955Z Updating files: 88% (17707/20121) 2025-12-04T08:57:46.0488594Z Updating files: 89% (17908/20121) 2025-12-04T08:57:46.0665072Z Updating files: 90% (18109/20121) 2025-12-04T08:57:46.0780881Z Updating files: 91% (18311/20121) 2025-12-04T08:57:46.0935533Z Updating files: 92% (18512/20121) 2025-12-04T08:57:46.1123206Z Updating files: 93% (18713/20121) 2025-12-04T08:57:46.1328520Z Updating files: 94% (18914/20121) 2025-12-04T08:57:46.1504106Z Updating files: 95% (19115/20121) 2025-12-04T08:57:46.1663595Z Updating files: 96% (19317/20121) 2025-12-04T08:57:46.1828509Z Updating files: 97% (19518/20121) 2025-12-04T08:57:46.2125262Z Updating files: 98% (19719/20121) 2025-12-04T08:57:46.2300328Z Updating files: 99% (19920/20121) 2025-12-04T08:57:46.2300851Z Updating files: 100% (20121/20121) 2025-12-04T08:57:46.2301849Z Updating files: 100% (20121/20121), done. 2025-12-04T08:57:46.2588751Z Note: switching to 'ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32'. 2025-12-04T08:57:46.2589192Z 2025-12-04T08:57:46.2589446Z You are in 'detached HEAD' state. You can look around, make experimental 2025-12-04T08:57:46.2590094Z changes and commit them, and you can discard any commits you make in this 2025-12-04T08:57:46.2590742Z state without impacting any branches by switching back to a branch. 2025-12-04T08:57:46.2591118Z 2025-12-04T08:57:46.2591357Z If you want to create a new branch to retain commits you create, you may 2025-12-04T08:57:46.2591954Z do so (now or later) by using -c with the switch command. Example: 2025-12-04T08:57:46.2592309Z 2025-12-04T08:57:46.2592439Z git switch -c 2025-12-04T08:57:46.2592665Z 2025-12-04T08:57:46.2592802Z Or undo this operation with: 2025-12-04T08:57:46.2593010Z 2025-12-04T08:57:46.2593112Z git switch - 2025-12-04T08:57:46.2593274Z 2025-12-04T08:57:46.2593566Z Turn off this advice by setting config variable advice.detachedHead to false 2025-12-04T08:57:46.2593981Z 2025-12-04T08:57:46.2594301Z HEAD is now at ffd9b0fb435 Resolve collective autotuning test failure on arm (#168919) 2025-12-04T08:57:46.2672464Z ##[endgroup] 2025-12-04T08:57:46.2672969Z ##[group]Setting up auth for fetching submodules 2025-12-04T08:57:46.2678825Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-12-04T08:57:46.2725462Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-12-04T08:57:46.2752293Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-12-04T08:57:46.2779065Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-12-04T08:57:46.2805240Z ##[endgroup] 2025-12-04T08:57:46.2805731Z ##[group]Fetching submodules 2025-12-04T08:57:46.2808433Z [command]/usr/bin/git submodule sync --recursive 2025-12-04T08:57:46.3145172Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-12-04T08:57:46.3473173Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-12-04T08:57:46.3474707Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-12-04T08:57:46.3477291Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-12-04T08:57:46.3479885Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-12-04T08:57:46.3482541Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-12-04T08:57:46.3485796Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-12-04T08:57:46.3488304Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-12-04T08:57:46.3491294Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-12-04T08:57:46.3494617Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-12-04T08:57:46.3498092Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-12-04T08:57:46.3501314Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-12-04T08:57:46.3504726Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-12-04T08:57:46.3508904Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-12-04T08:57:46.3512370Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-12-04T08:57:46.3516253Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-12-04T08:57:46.3522124Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-12-04T08:57:46.3528448Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-12-04T08:57:46.3532608Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-12-04T08:57:46.3537186Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:57:46.3541251Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-12-04T08:57:46.3545706Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-12-04T08:57:46.3550489Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-12-04T08:57:46.3555195Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-12-04T08:57:46.3560080Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-12-04T08:57:46.3564995Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-12-04T08:57:46.3569792Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-12-04T08:57:46.3574728Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-12-04T08:57:46.3579670Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-12-04T08:57:46.3585151Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-12-04T08:57:46.3590544Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-12-04T08:57:46.3596075Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-12-04T08:57:46.3601706Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-12-04T08:57:46.3607433Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-12-04T08:57:46.3614576Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-12-04T08:57:46.3620507Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-12-04T08:57:46.3626809Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-12-04T08:57:46.3633207Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-12-04T08:57:46.3665798Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-12-04T08:57:46.5832783Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-12-04T08:57:46.5834144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-12-04T08:57:46.5835202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-12-04T08:57:46.5836045Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-12-04T08:57:46.5837048Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-12-04T08:57:46.5969699Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-12-04T08:57:46.9751290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-12-04T08:57:46.9753660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-12-04T08:57:46.9756090Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-12-04T08:57:46.9757897Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-12-04T08:57:46.9759991Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-12-04T08:57:46.9762021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-12-04T08:57:46.9763974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-12-04T08:57:46.9765788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-12-04T08:57:46.9858104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-12-04T08:57:48.2469716Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-12-04T08:57:48.2471001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-12-04T08:57:48.2472256Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-12-04T08:57:48.2473965Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-12-04T08:57:48.2475698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-12-04T08:57:48.2477347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-12-04T08:57:48.2479354Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-12-04T08:57:48.2481261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-12-04T08:57:48.2482869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-12-04T08:57:48.2484534Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-12-04T08:57:48.2486567Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-12-04T08:57:48.2488251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-12-04T08:57:48.3471588Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-12-04T08:58:00.0291130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-12-04T08:58:00.0292056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-12-04T08:58:00.0292856Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-12-04T08:58:00.0293709Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-12-04T08:58:00.0294581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-12-04T08:58:00.0295441Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-12-04T08:58:00.0296314Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-12-04T08:58:00.1292916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-12-04T08:58:02.6807065Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-12-04T08:58:02.6946660Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-12-04T08:58:02.7049274Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-12-04T08:58:02.7309858Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-12-04T08:58:02.8189404Z Submodule path 'third_party/NVTX': checked out '3ebbc93ded7285963bff932c678fa367eb393ba6' 2025-12-04T08:58:02.8778733Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-12-04T08:58:03.6451836Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-12-04T08:58:03.8433319Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-12-04T08:58:03.8455646Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:58:03.8484489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-12-04T08:58:08.6025446Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-12-04T08:58:08.6272818Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-12-04T08:58:08.9981186Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T08:58:09.0511180Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246' 2025-12-04T08:58:09.1532460Z Submodule path 'third_party/cpuinfo': checked out 'f858c30bcb16f8effd5ff46996f0514539e17abc' 2025-12-04T08:58:09.2031152Z Submodule path 'third_party/cudnn_frontend': checked out '0b1577c8c83401237d601d0d0db5210506705396' 2025-12-04T08:58:09.8869381Z Submodule path 'third_party/cutlass': checked out 'f88806b1e31dfa579842638740216dd41fc6c588' 2025-12-04T08:58:10.0503280Z Submodule path 'third_party/fbgemm': checked out 'c0b988d39a9e47c794d699f29930ed4d7c7e13a4' 2025-12-04T08:58:10.0525863Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-12-04T08:58:10.0527607Z Submodule 'external/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:58:10.0529808Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:58:10.0532556Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-12-04T08:58:10.0535459Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-12-04T08:58:10.0538625Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:58:10.0541905Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-12-04T08:58:10.0572899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-12-04T08:58:11.1544735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-12-04T08:58:11.1545794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-12-04T08:58:11.1546774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-12-04T08:58:11.2546400Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-12-04T08:58:14.4654087Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-12-04T08:58:14.5655182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-12-04T08:58:17.0405809Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-12-04T08:58:17.4262066Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-12-04T08:58:17.5367248Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-12-04T08:58:18.2326996Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '98125ce499b0fdf7ffbe0e3052f5b8709f4840f8' 2025-12-04T08:58:18.2834299Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T08:58:18.2963866Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691' 2025-12-04T08:58:18.4154369Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-12-04T08:58:18.4910161Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-12-04T08:58:18.4930147Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:58:18.4932307Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:58:18.4963390Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-12-04T08:58:22.8457144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-12-04T08:58:23.0966884Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-12-04T08:58:23.6859789Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-12-04T08:58:23.8308599Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-12-04T08:58:23.8624062Z Submodule path 'third_party/fmt': checked out '407c905e45ad75fc29bf0f9bb7c5c2fd3475976f' 2025-12-04T08:58:23.9044502Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-12-04T08:58:23.9308581Z Submodule path 'third_party/gloo': checked out '54cbae0d3a67fa890b4c3d9ee162b7860315e341' 2025-12-04T08:58:23.9779255Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T08:58:23.9913154Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-12-04T08:58:23.9930949Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-12-04T08:58:23.9958330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-12-04T08:58:38.3985352Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-12-04T08:58:38.4198070Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-12-04T08:58:38.5102022Z Submodule path 'third_party/kineto': checked out '31f85df8fbd89c188f14ef10f1ec65379786b943' 2025-12-04T08:58:38.5123492Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:58:38.5126212Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:58:38.5127999Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:58:38.5153685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-12-04T08:58:39.1817814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-12-04T08:58:39.5761397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-12-04T08:58:39.6718239Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out 'd2ffe0a4e3acace628db49974246b66fc3e85fb1' 2025-12-04T08:58:39.6739594Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:58:39.6741058Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:58:39.6742441Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:58:39.6744892Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:58:39.6747980Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:58:39.6751785Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:58:39.6755033Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:58:39.6758231Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:58:39.6761675Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:58:39.6792183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-12-04T08:58:41.7128984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-12-04T08:58:41.7130431Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp'... 2025-12-04T08:58:41.7131840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-12-04T08:58:41.7133628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-12-04T08:58:41.7134963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-12-04T08:58:41.7136428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-12-04T08:58:41.7137749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-12-04T08:58:41.8129698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-12-04T08:58:46.7232336Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-12-04T08:58:46.7424942Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-12-04T08:58:46.7807662Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-12-04T08:58:46.7957611Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-12-04T08:58:46.7976573Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:58:46.8004401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-12-04T08:58:47.0817460Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-12-04T08:58:47.1011974Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-12-04T08:58:47.1492885Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T08:58:47.2551903Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-12-04T08:58:47.2727797Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-12-04T08:58:47.2907856Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp': checked out 'b1234816facfdda29845c46696a02998a4af115a' 2025-12-04T08:58:47.2926369Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:58:47.2928207Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:58:47.2956126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T08:58:49.2144026Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T08:58:49.4797680Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'd7ba35bbb649209c66e582d5a0244ba988a15159' 2025-12-04T08:58:49.5295928Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T08:58:49.5631739Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-12-04T08:58:49.6113677Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-12-04T08:58:49.6659236Z Submodule path 'third_party/kleidiai': checked out 'd7770c89632329a9914ef1a90289917597639cbe' 2025-12-04T08:58:49.7065014Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-12-04T08:58:49.8105999Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-12-04T08:58:50.2216211Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-12-04T08:58:50.2257500Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-12-04T08:58:50.2285842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-12-04T08:58:51.0205779Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-12-04T08:58:51.0926991Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-12-04T08:58:51.0947184Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:58:51.0948957Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:58:51.0951158Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:58:51.0954078Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:58:51.0957073Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:58:51.0959851Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:58:51.0962791Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:58:51.0965684Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:58:51.0993701Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-12-04T08:58:51.4629205Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-12-04T08:58:51.4630609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-12-04T08:58:51.4631854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-12-04T08:58:51.4633076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-12-04T08:58:51.5630101Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-12-04T08:58:52.0419687Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-12-04T08:58:58.2821049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-12-04T08:58:59.0174334Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-12-04T08:58:59.0594880Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-12-04T08:58:59.0774647Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-12-04T08:58:59.1853046Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-12-04T08:58:59.1993681Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-12-04T08:58:59.2152626Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-12-04T08:58:59.2314895Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-12-04T08:58:59.2331441Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:58:59.2333997Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:58:59.2363050Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-12-04T08:59:01.3588399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-12-04T08:59:01.6235082Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-12-04T08:59:01.6729427Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-12-04T08:59:02.1530361Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-12-04T08:59:02.1657582Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-12-04T08:59:02.4490629Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-12-04T08:59:02.4513877Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:59:02.4515815Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-12-04T08:59:02.4547856Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-12-04T08:59:02.9833684Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-12-04T08:59:03.3208321Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-12-04T08:59:03.3957404Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-12-04T08:59:03.4060589Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-12-04T08:59:03.4189648Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-12-04T08:59:03.4625791Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8' 2025-12-04T08:59:03.4927182Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-12-04T08:59:03.5389867Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-12-04T08:59:03.5669780Z Submodule path 'third_party/tensorpipe': checked out '2b4cd91092d335a697416b2a3cb398283246849d' 2025-12-04T08:59:03.5689579Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:59:03.5690919Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:59:03.5692663Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:59:03.5695467Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:59:03.5725668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-12-04T08:59:04.5228497Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-12-04T08:59:04.5229696Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-12-04T08:59:04.5339569Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-12-04T08:59:04.5941400Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-12-04T08:59:04.6107527Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-12-04T08:59:04.6878819Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-12-04T08:59:04.7183369Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-12-04T08:59:04.7198663Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:59:04.7226809Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-12-04T08:59:04.8989477Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-12-04T08:59:04.9028145Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-12-04T08:59:04.9346611Z Entering 'android/libs/fbjni' 2025-12-04T08:59:04.9389176Z Entering 'third_party/FP16' 2025-12-04T08:59:04.9436052Z Entering 'third_party/FXdiv' 2025-12-04T08:59:04.9482233Z Entering 'third_party/NNPACK' 2025-12-04T08:59:04.9525789Z Entering 'third_party/NVTX' 2025-12-04T08:59:04.9571867Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:59:04.9616411Z Entering 'third_party/XNNPACK' 2025-12-04T08:59:04.9676519Z Entering 'third_party/aiter' 2025-12-04T08:59:04.9723423Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:59:04.9776782Z Entering 'third_party/benchmark' 2025-12-04T08:59:04.9820876Z Entering 'third_party/composable_kernel' 2025-12-04T08:59:04.9884972Z Entering 'third_party/cpp-httplib' 2025-12-04T08:59:04.9926866Z Entering 'third_party/cpuinfo' 2025-12-04T08:59:04.9971785Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:59:05.0019276Z Entering 'third_party/cutlass' 2025-12-04T08:59:05.0076440Z Entering 'third_party/fbgemm' 2025-12-04T08:59:05.0123381Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:59:05.0167508Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:59:05.0217838Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:59:05.0261569Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:59:05.0313273Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:59:05.0365854Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:59:05.0406934Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:59:05.0455392Z Entering 'third_party/flash-attention' 2025-12-04T08:59:05.0501699Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:59:05.0551672Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:59:05.0615905Z Entering 'third_party/flatbuffers' 2025-12-04T08:59:05.0664340Z Entering 'third_party/fmt' 2025-12-04T08:59:05.0709049Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:59:05.0751047Z Entering 'third_party/gloo' 2025-12-04T08:59:05.0793962Z Entering 'third_party/googletest' 2025-12-04T08:59:05.0847663Z Entering 'third_party/ideep' 2025-12-04T08:59:05.0890120Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:59:05.0942978Z Entering 'third_party/ittapi' 2025-12-04T08:59:05.0987512Z Entering 'third_party/kineto' 2025-12-04T08:59:05.1029115Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:59:05.1070518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:59:05.1114201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:59:05.1158659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:59:05.1207373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:59:05.1248693Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:59:05.1293350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:59:05.1335097Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:59:05.1383705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:59:05.1427748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:59:05.1470686Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:59:05.1516594Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:05.1565234Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:05.1612690Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:59:05.1655153Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:59:05.1700598Z Entering 'third_party/kleidiai' 2025-12-04T08:59:05.1746696Z Entering 'third_party/mimalloc' 2025-12-04T08:59:05.1790791Z Entering 'third_party/nlohmann' 2025-12-04T08:59:05.1836296Z Entering 'third_party/onnx' 2025-12-04T08:59:05.1900684Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:59:05.1947031Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:59:05.1991254Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:59:05.2033558Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:59:05.2074914Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:59:05.2116115Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:59:05.2163375Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:59:05.2205214Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:59:05.2246992Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:59:05.2289499Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:05.2333496Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:05.2379543Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:59:05.2446357Z Entering 'third_party/pocketfft' 2025-12-04T08:59:05.2495674Z Entering 'third_party/protobuf' 2025-12-04T08:59:05.2544037Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:59:05.2586729Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:59:05.2630188Z Entering 'third_party/psimd' 2025-12-04T08:59:05.2674677Z Entering 'third_party/pthreadpool' 2025-12-04T08:59:05.2717578Z Entering 'third_party/pybind11' 2025-12-04T08:59:05.2763739Z Entering 'third_party/python-peachpy' 2025-12-04T08:59:05.2807478Z Entering 'third_party/sleef' 2025-12-04T08:59:05.2850335Z Entering 'third_party/tensorpipe' 2025-12-04T08:59:05.2894395Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:59:05.2934933Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:59:05.2978650Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:59:05.3025453Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:59:05.3067701Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:59:05.3123423Z ##[endgroup] 2025-12-04T08:59:05.3124515Z ##[group]Persisting credentials for submodules 2025-12-04T08:59:05.3130850Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-12-04T08:59:05.3449571Z Entering 'android/libs/fbjni' 2025-12-04T08:59:05.3507345Z Entering 'third_party/FP16' 2025-12-04T08:59:05.3566555Z Entering 'third_party/FXdiv' 2025-12-04T08:59:05.3624540Z Entering 'third_party/NNPACK' 2025-12-04T08:59:05.3685233Z Entering 'third_party/NVTX' 2025-12-04T08:59:05.3745508Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:59:05.3804710Z Entering 'third_party/XNNPACK' 2025-12-04T08:59:05.3877354Z Entering 'third_party/aiter' 2025-12-04T08:59:05.3934716Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:59:05.4006556Z Entering 'third_party/benchmark' 2025-12-04T08:59:05.4065869Z Entering 'third_party/composable_kernel' 2025-12-04T08:59:05.4131818Z Entering 'third_party/cpp-httplib' 2025-12-04T08:59:05.4189745Z Entering 'third_party/cpuinfo' 2025-12-04T08:59:05.4249426Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:59:05.4308280Z Entering 'third_party/cutlass' 2025-12-04T08:59:05.4379974Z Entering 'third_party/fbgemm' 2025-12-04T08:59:05.4443559Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:59:05.4501025Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:59:05.4572964Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:59:05.4633910Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:59:05.4700021Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:59:05.4767587Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:59:05.4825763Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:59:05.4892870Z Entering 'third_party/flash-attention' 2025-12-04T08:59:05.4949244Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:59:05.5015360Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:59:05.5089457Z Entering 'third_party/flatbuffers' 2025-12-04T08:59:05.5148760Z Entering 'third_party/fmt' 2025-12-04T08:59:05.5206566Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:59:05.5265510Z Entering 'third_party/gloo' 2025-12-04T08:59:05.5326353Z Entering 'third_party/googletest' 2025-12-04T08:59:05.5384665Z Entering 'third_party/ideep' 2025-12-04T08:59:05.5447186Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:59:05.5512215Z Entering 'third_party/ittapi' 2025-12-04T08:59:05.5571409Z Entering 'third_party/kineto' 2025-12-04T08:59:05.5626874Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:59:05.5684735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:59:05.5744846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:59:05.5808277Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:59:05.5866334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:59:05.5925205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:59:05.5988752Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:59:05.6046360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:59:05.6105271Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:59:05.6166804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:59:05.6225942Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:59:05.6284331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:05.6345299Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:05.6409561Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:59:05.6467431Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:59:05.6527059Z Entering 'third_party/kleidiai' 2025-12-04T08:59:05.6587708Z Entering 'third_party/mimalloc' 2025-12-04T08:59:05.6646425Z Entering 'third_party/nlohmann' 2025-12-04T08:59:05.6705048Z Entering 'third_party/onnx' 2025-12-04T08:59:05.6782664Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:59:05.6844602Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:59:05.6905753Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:59:05.6963067Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:59:05.7025410Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:59:05.7085796Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:59:05.7144714Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:59:05.7202612Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:59:05.7266306Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:59:05.7325161Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:05.7389470Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:05.7448203Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:59:05.7527485Z Entering 'third_party/pocketfft' 2025-12-04T08:59:05.7588844Z Entering 'third_party/protobuf' 2025-12-04T08:59:05.7648034Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:59:05.7708328Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:59:05.7768432Z Entering 'third_party/psimd' 2025-12-04T08:59:05.7825135Z Entering 'third_party/pthreadpool' 2025-12-04T08:59:05.7884183Z Entering 'third_party/pybind11' 2025-12-04T08:59:05.7943071Z Entering 'third_party/python-peachpy' 2025-12-04T08:59:05.7998636Z Entering 'third_party/sleef' 2025-12-04T08:59:05.8057331Z Entering 'third_party/tensorpipe' 2025-12-04T08:59:05.8113596Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:59:05.8170879Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:59:05.8226506Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:59:05.8285482Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:59:05.8341688Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:59:05.8414877Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-12-04T08:59:05.8731867Z Entering 'android/libs/fbjni' 2025-12-04T08:59:05.8785424Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T08:59:05.8803918Z Entering 'third_party/FP16' 2025-12-04T08:59:05.8857415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T08:59:05.8873994Z Entering 'third_party/FXdiv' 2025-12-04T08:59:05.8926696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T08:59:05.8945717Z Entering 'third_party/NNPACK' 2025-12-04T08:59:05.8998095Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T08:59:05.9018125Z Entering 'third_party/NVTX' 2025-12-04T08:59:05.9069622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T08:59:05.9092042Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:59:05.9145665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T08:59:05.9163624Z Entering 'third_party/XNNPACK' 2025-12-04T08:59:05.9216836Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T08:59:05.9252242Z Entering 'third_party/aiter' 2025-12-04T08:59:05.9305635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T08:59:05.9327634Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:59:05.9383433Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T08:59:05.9414036Z Entering 'third_party/benchmark' 2025-12-04T08:59:05.9469281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T08:59:05.9486876Z Entering 'third_party/composable_kernel' 2025-12-04T08:59:05.9541041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T08:59:05.9569002Z Entering 'third_party/cpp-httplib' 2025-12-04T08:59:05.9622138Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T08:59:05.9638818Z Entering 'third_party/cpuinfo' 2025-12-04T08:59:05.9692824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T08:59:05.9709626Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:59:05.9762569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T08:59:05.9781654Z Entering 'third_party/cutlass' 2025-12-04T08:59:05.9833042Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T08:59:05.9862800Z Entering 'third_party/fbgemm' 2025-12-04T08:59:05.9916710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T08:59:05.9939331Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:59:05.9989726Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T08:59:06.0007987Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:59:06.0060964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T08:59:06.0087761Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:59:06.0141642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T08:59:06.0157288Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:59:06.0210399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T08:59:06.0237164Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:59:06.0295463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T08:59:06.0311857Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:59:06.0365848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T08:59:06.0382746Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:59:06.0433826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T08:59:06.0456061Z Entering 'third_party/flash-attention' 2025-12-04T08:59:06.0509279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T08:59:06.0526642Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:59:06.0580134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T08:59:06.0606047Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:59:06.0658001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T08:59:06.0687233Z Entering 'third_party/flatbuffers' 2025-12-04T08:59:06.0739223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T08:59:06.0765244Z Entering 'third_party/fmt' 2025-12-04T08:59:06.0817159Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T08:59:06.0834033Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:59:06.0889226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T08:59:06.0906957Z Entering 'third_party/gloo' 2025-12-04T08:59:06.0958980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T08:59:06.0978961Z Entering 'third_party/googletest' 2025-12-04T08:59:06.1034534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:59:06.1053304Z Entering 'third_party/ideep' 2025-12-04T08:59:06.1106294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T08:59:06.1123975Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:59:06.1177007Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T08:59:06.1204250Z Entering 'third_party/ittapi' 2025-12-04T08:59:06.1258637Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T08:59:06.1274322Z Entering 'third_party/kineto' 2025-12-04T08:59:06.1327924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T08:59:06.1346822Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:59:06.1397801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T08:59:06.1415941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:59:06.1477313Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T08:59:06.1497921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:59:06.1555531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T08:59:06.1573870Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:59:06.1626063Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T08:59:06.1645092Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:59:06.1697805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T08:59:06.1712232Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:59:06.1769381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T08:59:06.1789299Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:59:06.1845397Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T08:59:06.1862329Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:59:06.1914530Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:59:06.1934565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:59:06.1985491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T08:59:06.2006520Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:59:06.2059854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T08:59:06.2075316Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:59:06.2129116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T08:59:06.2147610Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:06.2199158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T08:59:06.2220764Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:06.2273070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T08:59:06.2296930Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:59:06.2349838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T08:59:06.2367470Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:59:06.2420321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T08:59:06.2437263Z Entering 'third_party/kleidiai' 2025-12-04T08:59:06.2492792Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T08:59:06.2509278Z Entering 'third_party/mimalloc' 2025-12-04T08:59:06.2564650Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T08:59:06.2582587Z Entering 'third_party/nlohmann' 2025-12-04T08:59:06.2633892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T08:59:06.2654590Z Entering 'third_party/onnx' 2025-12-04T08:59:06.2707353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T08:59:06.2745227Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:59:06.2796061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T08:59:06.2817478Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:59:06.2870169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T08:59:06.2890107Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:59:06.2943353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T08:59:06.2963099Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:59:06.3014652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:59:06.3030440Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:59:06.3083108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T08:59:06.3100653Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:59:06.3152571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T08:59:06.3172789Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:59:06.3225331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T08:59:06.3241857Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:59:06.3294222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T08:59:06.3309473Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:59:06.3367265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T08:59:06.3384402Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:06.3444957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T08:59:06.3464766Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:06.3515714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T08:59:06.3533732Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:59:06.3587762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T08:59:06.3626340Z Entering 'third_party/pocketfft' 2025-12-04T08:59:06.3684119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T08:59:06.3701627Z Entering 'third_party/protobuf' 2025-12-04T08:59:06.3753479Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T08:59:06.3776937Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:59:06.3827071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T08:59:06.3844875Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:59:06.3898100Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:59:06.3915478Z Entering 'third_party/psimd' 2025-12-04T08:59:06.3970540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T08:59:06.3989422Z Entering 'third_party/pthreadpool' 2025-12-04T08:59:06.4041045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T08:59:06.4059046Z Entering 'third_party/pybind11' 2025-12-04T08:59:06.4109709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T08:59:06.4127394Z Entering 'third_party/python-peachpy' 2025-12-04T08:59:06.4182390Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T08:59:06.4203622Z Entering 'third_party/sleef' 2025-12-04T08:59:06.4256829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T08:59:06.4273776Z Entering 'third_party/tensorpipe' 2025-12-04T08:59:06.4325902Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T08:59:06.4344428Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:59:06.4396997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T08:59:06.4415074Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:59:06.4466808Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T08:59:06.4485077Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:59:06.4537696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T08:59:06.4553474Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:59:06.4606899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T08:59:06.4623163Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:59:06.4675557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T08:59:06.5283450Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-12-04T08:59:06.5602152Z Entering 'android/libs/fbjni' 2025-12-04T08:59:06.5646079Z Entering 'third_party/FP16' 2025-12-04T08:59:06.5689336Z Entering 'third_party/FXdiv' 2025-12-04T08:59:06.5732373Z Entering 'third_party/NNPACK' 2025-12-04T08:59:06.5777213Z Entering 'third_party/NVTX' 2025-12-04T08:59:06.5825353Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:59:06.5868858Z Entering 'third_party/XNNPACK' 2025-12-04T08:59:06.5928374Z Entering 'third_party/aiter' 2025-12-04T08:59:06.5974531Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:59:06.6025233Z Entering 'third_party/benchmark' 2025-12-04T08:59:06.6068865Z Entering 'third_party/composable_kernel' 2025-12-04T08:59:06.6124901Z Entering 'third_party/cpp-httplib' 2025-12-04T08:59:06.6169909Z Entering 'third_party/cpuinfo' 2025-12-04T08:59:06.6214805Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:59:06.6261328Z Entering 'third_party/cutlass' 2025-12-04T08:59:06.6315059Z Entering 'third_party/fbgemm' 2025-12-04T08:59:06.6368827Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:59:06.6411158Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:59:06.6462240Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:59:06.6505565Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:59:06.6557855Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:59:06.6605439Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:59:06.6648396Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:59:06.6694387Z Entering 'third_party/flash-attention' 2025-12-04T08:59:06.6742299Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:59:06.6791717Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:59:06.6847870Z Entering 'third_party/flatbuffers' 2025-12-04T08:59:06.6895044Z Entering 'third_party/fmt' 2025-12-04T08:59:06.6942239Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:59:06.6989047Z Entering 'third_party/gloo' 2025-12-04T08:59:06.7032182Z Entering 'third_party/googletest' 2025-12-04T08:59:06.7076106Z Entering 'third_party/ideep' 2025-12-04T08:59:06.7117485Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:59:06.7171341Z Entering 'third_party/ittapi' 2025-12-04T08:59:06.7214621Z Entering 'third_party/kineto' 2025-12-04T08:59:06.7259812Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:59:06.7303823Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:59:06.7348525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:59:06.7389763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:59:06.7433234Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:59:06.7473643Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:59:06.7517821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:59:06.7568243Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:59:06.7611370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:59:06.7655603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:59:06.7700060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:59:06.7744512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:06.7789384Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:06.7836237Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:59:06.7881796Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:59:06.7927757Z Entering 'third_party/kleidiai' 2025-12-04T08:59:06.7975824Z Entering 'third_party/mimalloc' 2025-12-04T08:59:06.8020762Z Entering 'third_party/nlohmann' 2025-12-04T08:59:06.8066751Z Entering 'third_party/onnx' 2025-12-04T08:59:06.8127672Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:59:06.8174868Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:59:06.8226339Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:59:06.8269210Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:59:06.8311077Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:59:06.8355564Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:59:06.8406811Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:59:06.8449129Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:59:06.8490937Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:59:06.8531755Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:06.8581095Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:06.8625407Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:59:06.8697944Z Entering 'third_party/pocketfft' 2025-12-04T08:59:06.8743609Z Entering 'third_party/protobuf' 2025-12-04T08:59:06.8788845Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:59:06.8829977Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:59:06.8874105Z Entering 'third_party/psimd' 2025-12-04T08:59:06.8918613Z Entering 'third_party/pthreadpool' 2025-12-04T08:59:06.8965021Z Entering 'third_party/pybind11' 2025-12-04T08:59:06.9009681Z Entering 'third_party/python-peachpy' 2025-12-04T08:59:06.9056862Z Entering 'third_party/sleef' 2025-12-04T08:59:06.9101636Z Entering 'third_party/tensorpipe' 2025-12-04T08:59:06.9145482Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:59:06.9188394Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:59:06.9229911Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:59:06.9271056Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:59:06.9312110Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:59:06.9378736Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-12-04T08:59:06.9699011Z Entering 'android/libs/fbjni' 2025-12-04T08:59:06.9744492Z Entering 'third_party/FP16' 2025-12-04T08:59:06.9787805Z Entering 'third_party/FXdiv' 2025-12-04T08:59:06.9831213Z Entering 'third_party/NNPACK' 2025-12-04T08:59:06.9874203Z Entering 'third_party/NVTX' 2025-12-04T08:59:06.9919091Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:59:06.9966499Z Entering 'third_party/XNNPACK' 2025-12-04T08:59:07.0025677Z Entering 'third_party/aiter' 2025-12-04T08:59:07.0069225Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:59:07.0121931Z Entering 'third_party/benchmark' 2025-12-04T08:59:07.0166014Z Entering 'third_party/composable_kernel' 2025-12-04T08:59:07.0218162Z Entering 'third_party/cpp-httplib' 2025-12-04T08:59:07.0262825Z Entering 'third_party/cpuinfo' 2025-12-04T08:59:07.0312796Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:59:07.0358749Z Entering 'third_party/cutlass' 2025-12-04T08:59:07.0414145Z Entering 'third_party/fbgemm' 2025-12-04T08:59:07.0462029Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:59:07.0505124Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:59:07.0556736Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:59:07.0606263Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:59:07.0658695Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:59:07.0704561Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:59:07.0746761Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:59:07.0795535Z Entering 'third_party/flash-attention' 2025-12-04T08:59:07.0842287Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:59:07.0895045Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:59:07.0955411Z Entering 'third_party/flatbuffers' 2025-12-04T08:59:07.1004320Z Entering 'third_party/fmt' 2025-12-04T08:59:07.1051991Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:59:07.1094965Z Entering 'third_party/gloo' 2025-12-04T08:59:07.1142387Z Entering 'third_party/googletest' 2025-12-04T08:59:07.1185980Z Entering 'third_party/ideep' 2025-12-04T08:59:07.1231771Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:59:07.1290031Z Entering 'third_party/ittapi' 2025-12-04T08:59:07.1331308Z Entering 'third_party/kineto' 2025-12-04T08:59:07.1376569Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:59:07.1420028Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:59:07.1468542Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:59:07.1510412Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:59:07.1555734Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:59:07.1596882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:59:07.1649009Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:59:07.1690866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:59:07.1735983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:59:07.1785553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:59:07.1834711Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:59:07.1876351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:07.1926619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:07.1975355Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:59:07.2018435Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:59:07.2068678Z Entering 'third_party/kleidiai' 2025-12-04T08:59:07.2112730Z Entering 'third_party/mimalloc' 2025-12-04T08:59:07.2167750Z Entering 'third_party/nlohmann' 2025-12-04T08:59:07.2212933Z Entering 'third_party/onnx' 2025-12-04T08:59:07.2273451Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:59:07.2318950Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:59:07.2367635Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:59:07.2411278Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:59:07.2457153Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:59:07.2502862Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:59:07.2548800Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:59:07.2589444Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:59:07.2635748Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:59:07.2676947Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:07.2726627Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:07.2772215Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:59:07.2834089Z Entering 'third_party/pocketfft' 2025-12-04T08:59:07.2886721Z Entering 'third_party/protobuf' 2025-12-04T08:59:07.2932017Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:59:07.2975548Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:59:07.3019321Z Entering 'third_party/psimd' 2025-12-04T08:59:07.3067415Z Entering 'third_party/pthreadpool' 2025-12-04T08:59:07.3109164Z Entering 'third_party/pybind11' 2025-12-04T08:59:07.3157155Z Entering 'third_party/python-peachpy' 2025-12-04T08:59:07.3202146Z Entering 'third_party/sleef' 2025-12-04T08:59:07.3245047Z Entering 'third_party/tensorpipe' 2025-12-04T08:59:07.3287938Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:59:07.3327477Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:59:07.3370962Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:59:07.3414540Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:59:07.3456058Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:59:07.3524322Z ##[endgroup] 2025-12-04T08:59:07.3557790Z [command]/usr/bin/git log -1 --format=%H 2025-12-04T08:59:07.3582826Z ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:59:07.3690348Z ##[group]Run cd "${GITHUB_WORKSPACE}" 2025-12-04T08:59:07.3690746Z cd "${GITHUB_WORKSPACE}" 2025-12-04T08:59:07.3691090Z # Clean stale submodule dirs 2025-12-04T08:59:07.3691454Z if [ -z "${NO_SUDO}" ]; then 2025-12-04T08:59:07.3691895Z  sudo git submodule foreach --recursive git clean -ffdx 2025-12-04T08:59:07.3692322Z else 2025-12-04T08:59:07.3692662Z  git submodule foreach --recursive git clean -ffdx 2025-12-04T08:59:07.3693071Z fi 2025-12-04T08:59:07.3701197Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:07.3701617Z env: 2025-12-04T08:59:07.3702003Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:07.3702281Z NO_SUDO: true 2025-12-04T08:59:07.3702630Z ##[endgroup] 2025-12-04T08:59:07.4041155Z Entering 'android/libs/fbjni' 2025-12-04T08:59:07.4074737Z Entering 'third_party/FP16' 2025-12-04T08:59:07.4108571Z Entering 'third_party/FXdiv' 2025-12-04T08:59:07.4141307Z Entering 'third_party/NNPACK' 2025-12-04T08:59:07.4177742Z Entering 'third_party/NVTX' 2025-12-04T08:59:07.4218927Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T08:59:07.4253110Z Entering 'third_party/XNNPACK' 2025-12-04T08:59:07.4383146Z Entering 'third_party/aiter' 2025-12-04T08:59:07.4426845Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T08:59:07.4531467Z Entering 'third_party/benchmark' 2025-12-04T08:59:07.4565085Z Entering 'third_party/composable_kernel' 2025-12-04T08:59:07.4679222Z Entering 'third_party/cpp-httplib' 2025-12-04T08:59:07.4714353Z Entering 'third_party/cpuinfo' 2025-12-04T08:59:07.4752489Z Entering 'third_party/cudnn_frontend' 2025-12-04T08:59:07.4792544Z Entering 'third_party/cutlass' 2025-12-04T08:59:07.4894773Z Entering 'third_party/fbgemm' 2025-12-04T08:59:07.4957041Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T08:59:07.4989005Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T08:59:07.5100840Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T08:59:07.5145945Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T08:59:07.5247741Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T08:59:07.5282933Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T08:59:07.5313849Z Entering 'third_party/fbgemm/external/json' 2025-12-04T08:59:07.5359171Z Entering 'third_party/flash-attention' 2025-12-04T08:59:07.5402258Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T08:59:07.5497750Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T08:59:07.5596190Z Entering 'third_party/flatbuffers' 2025-12-04T08:59:07.5664698Z Entering 'third_party/fmt' 2025-12-04T08:59:07.5699806Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T08:59:07.5733293Z Entering 'third_party/gloo' 2025-12-04T08:59:07.5768496Z Entering 'third_party/googletest' 2025-12-04T08:59:07.5804846Z Entering 'third_party/ideep' 2025-12-04T08:59:07.5835203Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T08:59:07.5922969Z Entering 'third_party/ittapi' 2025-12-04T08:59:07.5957909Z Entering 'third_party/kineto' 2025-12-04T08:59:07.5994006Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T08:59:07.6031373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T08:59:07.6075425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T08:59:07.6112032Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T08:59:07.6146032Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T08:59:07.6177151Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T08:59:07.6210257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T08:59:07.6241111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T08:59:07.6275642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T08:59:07.6325323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T08:59:07.6357983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T08:59:07.6394262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:07.6444168Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:07.6489296Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T08:59:07.6522870Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T08:59:07.6558656Z Entering 'third_party/kleidiai' 2025-12-04T08:59:07.6601132Z Entering 'third_party/mimalloc' 2025-12-04T08:59:07.6635550Z Entering 'third_party/nlohmann' 2025-12-04T08:59:07.6683096Z Entering 'third_party/onnx' 2025-12-04T08:59:07.6982485Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T08:59:07.7020484Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T08:59:07.7074805Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T08:59:07.7109923Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T08:59:07.7144696Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T08:59:07.7175709Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T08:59:07.7219653Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T08:59:07.7251937Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T08:59:07.7284702Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T08:59:07.7324309Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T08:59:07.7372498Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T08:59:07.7407984Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T08:59:07.7649107Z Entering 'third_party/pocketfft' 2025-12-04T08:59:07.7681166Z Entering 'third_party/protobuf' 2025-12-04T08:59:07.7758019Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T08:59:07.7794061Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T08:59:07.7831268Z Entering 'third_party/psimd' 2025-12-04T08:59:07.7863475Z Entering 'third_party/pthreadpool' 2025-12-04T08:59:07.7896135Z Entering 'third_party/pybind11' 2025-12-04T08:59:07.7930521Z Entering 'third_party/python-peachpy' 2025-12-04T08:59:07.7964305Z Entering 'third_party/sleef' 2025-12-04T08:59:07.7998457Z Entering 'third_party/tensorpipe' 2025-12-04T08:59:07.8033522Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T08:59:07.8067618Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T08:59:07.8098512Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T08:59:07.8133402Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T08:59:07.8164996Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T08:59:07.8336419Z Prepare all required actions 2025-12-04T08:59:07.8336937Z Getting action download info 2025-12-04T08:59:07.9899584Z ##[group]Run ./.github/actions/setup-linux 2025-12-04T08:59:07.9899897Z env: 2025-12-04T08:59:07.9900123Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:07.9900389Z ##[endgroup] 2025-12-04T08:59:07.9941893Z ##[group]Run set -euo pipefail 2025-12-04T08:59:07.9942242Z set -euo pipefail 2025-12-04T08:59:07.9942547Z function get_ec2_metadata() { 2025-12-04T08:59:07.9942940Z  # Pulled from instance metadata endpoint for EC2 2025-12-04T08:59:07.9943585Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-12-04T08:59:07.9944181Z  category=$1 2025-12-04T08:59:07.9944555Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-12-04T08:59:07.9944999Z  runner_name_str=i-0452b6a3e0f4cdcdb 2025-12-04T08:59:07.9945399Z  if [[ -f /.inarc ]]; then 2025-12-04T08:59:07.9945749Z  echo "ARC Runner, no info on ec2 metadata" 2025-12-04T08:59:07.9946153Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-12-04T08:59:07.9946638Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-12-04T08:59:07.9947087Z  else 2025-12-04T08:59:07.9948310Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-12-04T08:59:07.9949391Z  fi 2025-12-04T08:59:07.9949643Z } 2025-12-04T08:59:07.9950103Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-12-04T08:59:07.9950588Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-12-04T08:59:07.9951151Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-12-04T08:59:07.9951640Z echo "system info $(uname -a)" 2025-12-04T08:59:07.9958823Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:07.9959261Z env: 2025-12-04T08:59:07.9959509Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:07.9959899Z ##[endgroup] 2025-12-04T08:59:08.0109208Z ami-id: ami-08982f1c5bf93d976 2025-12-04T08:59:08.0229102Z instance-id: i-0452b6a3e0f4cdcdb 2025-12-04T08:59:08.0335761Z instance-type: g4dn.12xlarge 2025-12-04T08:59:08.0347754Z system info Linux ip-10-1-50-12.ec2.internal 6.1.150-174.273.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 9 12:21:26 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-12-04T08:59:08.0367403Z ##[group]Run if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T08:59:08.0367905Z if [ -f /usr/bin/nvidia-smi ]; then nvidia-smi; fi 2025-12-04T08:59:08.0373670Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:08.0394716Z env: 2025-12-04T08:59:08.0394967Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:08.0395291Z ##[endgroup] 2025-12-04T08:59:10.1149113Z Thu Dec 4 08:59:10 2025 2025-12-04T08:59:10.1149669Z +-----------------------------------------------------------------------------------------+ 2025-12-04T08:59:10.1150317Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T08:59:10.1150943Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T08:59:10.1151575Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T08:59:10.1152261Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T08:59:10.1152809Z | | | MIG M. | 2025-12-04T08:59:10.1153246Z |=========================================+========================+======================| 2025-12-04T08:59:10.1528663Z | 0 Tesla T4 Off | 00000000:00:1B.0 Off | 0 | 2025-12-04T08:59:10.1529495Z | N/A 34C P0 27W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T08:59:10.1529998Z | | | N/A | 2025-12-04T08:59:10.1530501Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T08:59:10.1531059Z | 1 Tesla T4 Off | 00000000:00:1C.0 Off | 0 | 2025-12-04T08:59:10.1531580Z | N/A 35C P0 25W / 70W | 0MiB / 15360MiB | 2% Default | 2025-12-04T08:59:10.1532056Z | | | N/A | 2025-12-04T08:59:10.1532562Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T08:59:10.1533111Z | 2 Tesla T4 Off | 00000000:00:1D.0 Off | 0 | 2025-12-04T08:59:10.1533635Z | N/A 34C P0 25W / 70W | 0MiB / 15360MiB | 4% Default | 2025-12-04T08:59:10.1534110Z | | | N/A | 2025-12-04T08:59:10.1534604Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T08:59:10.1535137Z | 3 Tesla T4 Off | 00000000:00:1E.0 Off | 0 | 2025-12-04T08:59:10.1535668Z | N/A 36C P0 25W / 70W | 0MiB / 15360MiB | 4% Default | 2025-12-04T08:59:10.1536140Z | | | N/A | 2025-12-04T08:59:10.1536738Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T08:59:10.1537221Z 2025-12-04T08:59:10.1537431Z +-----------------------------------------------------------------------------------------+ 2025-12-04T08:59:10.1537970Z | Processes: | 2025-12-04T08:59:10.1538519Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T08:59:10.1539024Z | ID ID Usage | 2025-12-04T08:59:10.1539425Z |=========================================================================================| 2025-12-04T08:59:10.1552844Z | No running processes found | 2025-12-04T08:59:10.1553453Z +-----------------------------------------------------------------------------------------+ 2025-12-04T08:59:11.7816007Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:59:11.7817272Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T08:59:11.7824515Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:11.7824955Z env: 2025-12-04T08:59:11.7825215Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:11.7825502Z ##[endgroup] 2025-12-04T08:59:11.7883649Z ##[group]Run if systemctl is-active --quiet docker; then 2025-12-04T08:59:11.7884105Z if systemctl is-active --quiet docker; then 2025-12-04T08:59:11.7884509Z  echo "Docker daemon is running..."; 2025-12-04T08:59:11.7884847Z else 2025-12-04T08:59:11.7885204Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-12-04T08:59:11.7885818Z fi 2025-12-04T08:59:11.7891463Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:11.7891897Z env: 2025-12-04T08:59:11.7892132Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:11.7892400Z ##[endgroup] 2025-12-04T08:59:11.7974892Z Docker daemon is running... 2025-12-04T08:59:11.8017534Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T08:59:11.8017856Z with: 2025-12-04T08:59:11.8018077Z shell: bash 2025-12-04T08:59:11.8018473Z timeout_minutes: 5 2025-12-04T08:59:11.8018741Z max_attempts: 3 2025-12-04T08:59:11.8018987Z retry_wait_seconds: 30 2025-12-04T08:59:11.8021522Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-12-04T08:59:11.8024471Z polling_interval_seconds: 1 2025-12-04T08:59:11.8024838Z warning_on_retry: true 2025-12-04T08:59:11.8025143Z continue_on_error: false 2025-12-04T08:59:11.8025422Z env: 2025-12-04T08:59:11.8025665Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:11.8025969Z AWS_RETRY_MODE: standard 2025-12-04T08:59:11.8026255Z AWS_MAX_ATTEMPTS: 5 2025-12-04T08:59:11.8026549Z AWS_DEFAULT_REGION: us-east-1 2025-12-04T08:59:11.8026864Z ##[endgroup] 2025-12-04T08:59:12.9607459Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:59:12.9608160Z Configure a credential helper to remove this warning. See 2025-12-04T08:59:12.9608812Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:59:12.9609251Z 2025-12-04T08:59:12.9609351Z Login Succeeded 2025-12-04T08:59:13.4939015Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:59:13.4941448Z Configure a credential helper to remove this warning. See 2025-12-04T08:59:13.4943117Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:59:13.4944266Z 2025-12-04T08:59:13.4944512Z Login Succeeded 2025-12-04T08:59:13.8913723Z Command completed after 1 attempt(s). 2025-12-04T08:59:13.8968184Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:59:13.8968745Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:59:13.8969259Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T08:59:13.8977052Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:13.8977470Z env: 2025-12-04T08:59:13.8977719Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:13.8978022Z ##[endgroup] 2025-12-04T08:59:13.9056556Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T08:59:13.9057143Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T08:59:13.9057595Z # shellcheck disable=SC2046 2025-12-04T08:59:13.9057946Z docker stop $(docker ps -q) || true 2025-12-04T08:59:13.9058306Z # Prune all of the docker images 2025-12-04T08:59:13.9058643Z docker system prune -af 2025-12-04T08:59:13.9064664Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:13.9065072Z env: 2025-12-04T08:59:13.9065303Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:13.9065573Z ##[endgroup] 2025-12-04T08:59:13.9319153Z "docker stop" requires at least 1 argument. 2025-12-04T08:59:13.9319638Z See 'docker stop --help'. 2025-12-04T08:59:13.9319957Z 2025-12-04T08:59:13.9320144Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-12-04T08:59:13.9320468Z 2025-12-04T08:59:13.9320591Z Stop one or more running containers 2025-12-04T08:59:13.9621520Z Total reclaimed space: 0B 2025-12-04T08:59:13.9822162Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-12-04T08:59:13.9822694Z with: 2025-12-04T08:59:13.9823802Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:13.9825020Z use-custom-docker-registry: true 2025-12-04T08:59:13.9825386Z docker-build-dir: .ci/docker 2025-12-04T08:59:13.9825727Z docker-build-script: ./build.sh 2025-12-04T08:59:13.9826059Z working-directory: . 2025-12-04T08:59:13.9826464Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:13.9826928Z force-push: false 2025-12-04T08:59:13.9827273Z env: 2025-12-04T08:59:13.9827533Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:13.9827835Z ##[endgroup] 2025-12-04T08:59:13.9847293Z ##[group]Run set -ex 2025-12-04T08:59:13.9847591Z set -ex 2025-12-04T08:59:13.9847824Z  2025-12-04T08:59:13.9848286Z # If the docker build directory or the build script doesn't exist, the action will 2025-12-04T08:59:13.9848998Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-12-04T08:59:13.9849588Z # job could then download the pre-built image as usual 2025-12-04T08:59:13.9850310Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-12-04T08:59:13.9850972Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9851314Z else 2025-12-04T08:59:13.9851585Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9852212Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9852646Z  2025-12-04T08:59:13.9853255Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-12-04T08:59:13.9853966Z  exit 0 2025-12-04T08:59:13.9854336Z fi 2025-12-04T08:59:13.9854563Z  2025-12-04T08:59:13.9854930Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-12-04T08:59:13.9855595Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-12-04T08:59:13.9856171Z  # use it as it is, but first let's extract the tag 2025-12-04T08:59:13.9856698Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-12-04T08:59:13.9857255Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9857781Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9858234Z else 2025-12-04T08:59:13.9858524Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-12-04T08:59:13.9858950Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-12-04T08:59:13.9859375Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-12-04T08:59:13.9859751Z  fi 2025-12-04T08:59:13.9860248Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-12-04T08:59:13.9860911Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9861627Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9862404Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9862876Z fi 2025-12-04T08:59:13.9869149Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:13.9869591Z env: 2025-12-04T08:59:13.9869845Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:13.9870142Z REPO_NAME: pytorch 2025-12-04T08:59:13.9871260Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:13.9872309Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T08:59:13.9872643Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-12-04T08:59:13.9873076Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:13.9873549Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-12-04T08:59:13.9873894Z CUSTOM_TAG_PREFIX: 2025-12-04T08:59:13.9874164Z ##[endgroup] 2025-12-04T08:59:13.9898807Z + [[ -d .ci/docker ]] 2025-12-04T08:59:13.9899155Z + [[ -f .ci/docker/./build.sh ]] 2025-12-04T08:59:13.9899538Z + [[ true == \t\r\u\e ]] 2025-12-04T08:59:13.9899834Z + echo skip=false 2025-12-04T08:59:13.9901041Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-12-04T08:59:13.9906580Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:13.9907819Z ++ awk -F '[:,]' '{print $2}' 2025-12-04T08:59:13.9929777Z + DOCKER_TAG=pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:13.9930847Z + echo docker-tag=pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:13.9932251Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:13.9956540Z ##[group]Run set +e 2025-12-04T08:59:13.9956889Z set +e 2025-12-04T08:59:13.9957141Z set -x 2025-12-04T08:59:13.9957395Z  2025-12-04T08:59:13.9957637Z login() { 2025-12-04T08:59:13.9958179Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T08:59:13.9958788Z } 2025-12-04T08:59:13.9959040Z  2025-12-04T08:59:13.9959392Z retry () { 2025-12-04T08:59:13.9959806Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T08:59:13.9960153Z } 2025-12-04T08:59:13.9960364Z  2025-12-04T08:59:13.9960621Z retry login "${DOCKER_REGISTRY}" 2025-12-04T08:59:13.9960960Z  2025-12-04T08:59:13.9961196Z START_TIME=$(date +%s) 2025-12-04T08:59:13.9961506Z # Wait up to 120 minutes 2025-12-04T08:59:13.9961903Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-12-04T08:59:13.9962448Z  # Check if image already exists, if it does then skip building it 2025-12-04T08:59:13.9962977Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-12-04T08:59:13.9963374Z  exit 0 2025-12-04T08:59:13.9963624Z  fi 2025-12-04T08:59:13.9963841Z  2025-12-04T08:59:13.9964264Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-12-04T08:59:13.9965007Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-12-04T08:59:13.9965741Z  # latter, it will wait for the Docker images to become available before continuing 2025-12-04T08:59:13.9966305Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-12-04T08:59:13.9966747Z  # It's a Docker build job, let's build the image 2025-12-04T08:59:13.9967127Z  break 2025-12-04T08:59:13.9967383Z  else 2025-12-04T08:59:13.9967743Z  # It's a regular build job, wait for the image to become available 2025-12-04T08:59:13.9968196Z  sleep 300 2025-12-04T08:59:13.9968463Z  fi 2025-12-04T08:59:13.9968684Z done 2025-12-04T08:59:13.9991526Z  2025-12-04T08:59:13.9991958Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-12-04T08:59:13.9992807Z # be empty. The default action would be to continue rebuild the image 2025-12-04T08:59:13.9993415Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-12-04T08:59:13.9993938Z  # if we're on the base branch then use the parent commit 2025-12-04T08:59:13.9994396Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-12-04T08:59:13.9994743Z else 2025-12-04T08:59:13.9995116Z  # otherwise we're on a PR, so use the most recent base commit 2025-12-04T08:59:13.9995664Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-12-04T08:59:13.9996063Z fi 2025-12-04T08:59:13.9996295Z  2025-12-04T08:59:13.9996544Z if [[ -z "${MERGE_BASE}" ]]; then 2025-12-04T08:59:13.9996939Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:13.9997297Z  2025-12-04T08:59:13.9997827Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-12-04T08:59:13.9998457Z  exit 0 2025-12-04T08:59:13.9998695Z fi 2025-12-04T08:59:13.9998916Z  2025-12-04T08:59:13.9999250Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-12-04T08:59:14.0000238Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-12-04T08:59:14.0000828Z  exit 1 2025-12-04T08:59:14.0001049Z fi 2025-12-04T08:59:14.0001257Z  2025-12-04T08:59:14.0001608Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-12-04T08:59:14.0002287Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-12-04T08:59:14.0002884Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-12-04T08:59:14.0003584Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-12-04T08:59:14.0004371Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-12-04T08:59:14.0004910Z fi 2025-12-04T08:59:14.0005123Z  2025-12-04T08:59:14.0005378Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-12-04T08:59:14.0010834Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:14.0011221Z env: 2025-12-04T08:59:14.0011433Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:14.0011718Z DOCKER_BUILD_DIR: .ci/docker 2025-12-04T08:59:14.0012076Z BASE_REVISION: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T08:59:14.0013034Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:14.0014195Z DOCKER_TAG: pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:14.0014895Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:14.0015309Z DOCKER_PUSH: 2025-12-04T08:59:14.0015550Z ##[endgroup] 2025-12-04T08:59:14.0039784Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:14.0040330Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:14.0043382Z + aws ecr get-login-password --region us-east-1 2025-12-04T08:59:14.0044688Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:14.5360825Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:59:14.5362594Z Configure a credential helper to remove this warning. See 2025-12-04T08:59:14.5363730Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:59:14.5364188Z 2025-12-04T08:59:14.5364439Z Login Succeeded 2025-12-04T08:59:14.5383262Z ++ date +%s 2025-12-04T08:59:14.5391656Z + START_TIME=1764838754 2025-12-04T08:59:14.5395055Z ++ date +%s 2025-12-04T08:59:14.5404424Z + [[ 1764831554 -lt 1764838754 ]] 2025-12-04T08:59:14.5405463Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:14.7507446Z { 2025-12-04T08:59:14.7507810Z "schemaVersion": 2, 2025-12-04T08:59:14.7508489Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-12-04T08:59:14.7509068Z "config": { 2025-12-04T08:59:14.7509454Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-12-04T08:59:14.7509921Z "size": 34864, 2025-12-04T08:59:14.7510378Z "digest": "sha256:add7313791033822205cdb3cf32096534b2cfaa4855bd48119b59000bfe00301" 2025-12-04T08:59:14.7510930Z }, 2025-12-04T08:59:14.7511161Z "layers": [ 2025-12-04T08:59:14.7511388Z { 2025-12-04T08:59:14.7511767Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7512253Z "size": 30447951, 2025-12-04T08:59:14.7512746Z "digest": "sha256:63e5bc7682b85ae57a1221210f64d62e7a90b0a30f19af4ca734b8242ae49d63" 2025-12-04T08:59:14.7513407Z }, 2025-12-04T08:59:14.7513617Z { 2025-12-04T08:59:14.7513965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7514431Z "size": 1554, 2025-12-04T08:59:14.7514884Z "digest": "sha256:0678d56345c994444b77bb70b1177189d23e794748b1d75ffc45d227c7dea94a" 2025-12-04T08:59:14.7515401Z }, 2025-12-04T08:59:14.7515601Z { 2025-12-04T08:59:14.7515961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7516428Z "size": 313275661, 2025-12-04T08:59:14.7516899Z "digest": "sha256:45f5c9ddfce78349dff3d5edfbaa0310ae17311f66abdcd7e00fa21b500e801c" 2025-12-04T08:59:14.7517440Z }, 2025-12-04T08:59:14.7517649Z { 2025-12-04T08:59:14.7517994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7518459Z "size": 787, 2025-12-04T08:59:14.7518919Z "digest": "sha256:086b1df51ac1162d9c45698e9dfaf91c6c222c8bd9ab01797ac8f9344bc8044f" 2025-12-04T08:59:14.7519447Z }, 2025-12-04T08:59:14.7519895Z { 2025-12-04T08:59:14.7520260Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7520716Z "size": 106, 2025-12-04T08:59:14.7521188Z "digest": "sha256:fe8a7b64bf98352f89057bcba66beef2fb44cc05fbd3606abccd8e86cf476234" 2025-12-04T08:59:14.7521732Z }, 2025-12-04T08:59:14.7521945Z { 2025-12-04T08:59:14.7522441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7522918Z "size": 703, 2025-12-04T08:59:14.7523373Z "digest": "sha256:7680723e9a578033dd106b45784c639f06cc8adb1f5239ec513d9de01087c1af" 2025-12-04T08:59:14.7524097Z }, 2025-12-04T08:59:14.7524483Z { 2025-12-04T08:59:14.7524859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7525323Z "size": 1216, 2025-12-04T08:59:14.7525797Z "digest": "sha256:9c5027aeeb4e3101f48c1d2e400c387110e1009e42497ee801f1b4b7f7edb5c0" 2025-12-04T08:59:14.7526338Z }, 2025-12-04T08:59:14.7526545Z { 2025-12-04T08:59:14.7526922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7527397Z "size": 483, 2025-12-04T08:59:14.7527858Z "digest": "sha256:9a56521103600bd37a1e7c1191b5136c2d738c092f8a6701499f7068a32c2628" 2025-12-04T08:59:14.7528377Z }, 2025-12-04T08:59:14.7528593Z { 2025-12-04T08:59:14.7528967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7529550Z "size": 110361875, 2025-12-04T08:59:14.7530012Z "digest": "sha256:375c4427e9141269458333b1463fdb219e736fd6231ec1c56c625c48437ace77" 2025-12-04T08:59:14.7530524Z }, 2025-12-04T08:59:14.7530827Z { 2025-12-04T08:59:14.7531178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7531637Z "size": 4961, 2025-12-04T08:59:14.7532074Z "digest": "sha256:a86faaa7dbdd70e678e5ea20072637ee42618921ca8f80ca089f789325d4b0c2" 2025-12-04T08:59:14.7532589Z }, 2025-12-04T08:59:14.7532790Z { 2025-12-04T08:59:14.7533247Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7533707Z "size": 1755, 2025-12-04T08:59:14.7534148Z "digest": "sha256:fb7848686804957915d98f8655ef6da0fe4c521b50a82aefdebf475983505a15" 2025-12-04T08:59:14.7534658Z }, 2025-12-04T08:59:14.7534853Z { 2025-12-04T08:59:14.7535210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7535666Z "size": 724, 2025-12-04T08:59:14.7536093Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T08:59:14.7536602Z }, 2025-12-04T08:59:14.7536806Z { 2025-12-04T08:59:14.7537141Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7537592Z "size": 543, 2025-12-04T08:59:14.7538027Z "digest": "sha256:79dc80f426b29d4ae9157b967050b03e66aa0c4b1295b944a1dd70106be87066" 2025-12-04T08:59:14.7538524Z }, 2025-12-04T08:59:14.7538728Z { 2025-12-04T08:59:14.7539080Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7539528Z "size": 3185190117, 2025-12-04T08:59:14.7540003Z "digest": "sha256:a13fcc1b90bb9c251ebe7ef2a03c4cb3afa1c8bdafe84f5f85136773059a3735" 2025-12-04T08:59:14.7540530Z }, 2025-12-04T08:59:14.7540737Z { 2025-12-04T08:59:14.7541077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7541523Z "size": 32, 2025-12-04T08:59:14.7541966Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7542474Z }, 2025-12-04T08:59:14.7542680Z { 2025-12-04T08:59:14.7543036Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7543485Z "size": 396, 2025-12-04T08:59:14.7543912Z "digest": "sha256:549db4d6c618ecd9534658a233e3c90508f82d8735f965c2786b2eaa078869e5" 2025-12-04T08:59:14.7544422Z }, 2025-12-04T08:59:14.7544629Z { 2025-12-04T08:59:14.7544970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7545437Z "size": 236860, 2025-12-04T08:59:14.7545981Z "digest": "sha256:5c63528cb580001e65104f4cb0809bf0673a00f989a7db42fd6d86aa1ec27cee" 2025-12-04T08:59:14.7546482Z }, 2025-12-04T08:59:14.7546693Z { 2025-12-04T08:59:14.7547048Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7547563Z "size": 231, 2025-12-04T08:59:14.7548192Z "digest": "sha256:75bd83b989a44e4d4119a3f972891025eb0e9ce95cfbe4a0ca5cdbe7130028d6" 2025-12-04T08:59:14.7548746Z }, 2025-12-04T08:59:14.7548966Z { 2025-12-04T08:59:14.7549328Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7549809Z "size": 3043497, 2025-12-04T08:59:14.7550289Z "digest": "sha256:de6e78970f517178cb91f36cd02bd9ca7b72a08fb82a0f9007516026f258c035" 2025-12-04T08:59:14.7550822Z }, 2025-12-04T08:59:14.7551137Z + exit 0 2025-12-04T08:59:14.7551365Z { 2025-12-04T08:59:14.7551726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7552213Z "size": 1472, 2025-12-04T08:59:14.7552701Z "digest": "sha256:e13ed7c7e4736e81dc21af755b3363eb26e4d3b2f1ca988dfe65effa47d8fa42" 2025-12-04T08:59:14.7553249Z }, 2025-12-04T08:59:14.7553449Z { 2025-12-04T08:59:14.7553818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7554293Z "size": 481, 2025-12-04T08:59:14.7554749Z "digest": "sha256:6e2949bcb74152577a0f20c38bcb6dd80f5e68427e3e531a80e08c9ecc73a979" 2025-12-04T08:59:14.7555296Z }, 2025-12-04T08:59:14.7555514Z { 2025-12-04T08:59:14.7555873Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7556352Z "size": 202, 2025-12-04T08:59:14.7556826Z "digest": "sha256:14d69d9aaec70287efd2fd35c4f93e43a29a4098458cc9fca1c93f02ad7356cb" 2025-12-04T08:59:14.7557364Z }, 2025-12-04T08:59:14.7557581Z { 2025-12-04T08:59:14.7557953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7558418Z "size": 607, 2025-12-04T08:59:14.7558980Z "digest": "sha256:5c02769dd8e5bba2f7f5fd84bde9595fcb3bdbffcae497503fa846f9b5e78bf5" 2025-12-04T08:59:14.7559541Z }, 2025-12-04T08:59:14.7559757Z { 2025-12-04T08:59:14.7560216Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7560666Z "size": 7889619584, 2025-12-04T08:59:14.7561128Z "digest": "sha256:35041ce524ac4afec40ecd73b1393c830614f1f79d43a6439767a6c7d5b7027b" 2025-12-04T08:59:14.7561626Z }, 2025-12-04T08:59:14.7561827Z { 2025-12-04T08:59:14.7562174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7562610Z "size": 830, 2025-12-04T08:59:14.7563051Z "digest": "sha256:2fa92dc5885e080e049ceb4139288b6c0e39fab34256945708b08ea55a1f7a0b" 2025-12-04T08:59:14.7563562Z }, 2025-12-04T08:59:14.7563755Z { 2025-12-04T08:59:14.7564104Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7564556Z "size": 33451739, 2025-12-04T08:59:14.7565002Z "digest": "sha256:2b85eafbd92a0e70a0a70154ad8bf4584095e576d95873368f30373f5966714a" 2025-12-04T08:59:14.7565515Z }, 2025-12-04T08:59:14.7565716Z { 2025-12-04T08:59:14.7566062Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7566498Z "size": 104, 2025-12-04T08:59:14.7566950Z "digest": "sha256:ff755a4ddad7880f23c6b767d432d6f1eafdb62b3ea18f8a98e22c441c099fcb" 2025-12-04T08:59:14.7567474Z }, 2025-12-04T08:59:14.7567664Z { 2025-12-04T08:59:14.7568012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7568460Z "size": 1496, 2025-12-04T08:59:14.7568885Z "digest": "sha256:09eb41bdf42d8605b57b2363348154140904dec914b34a67298b82122bfce2b3" 2025-12-04T08:59:14.7569388Z }, 2025-12-04T08:59:14.7569591Z { 2025-12-04T08:59:14.7569927Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7570376Z "size": 458787828, 2025-12-04T08:59:14.7570836Z "digest": "sha256:11ede4d59e935e62f41b33220fe871794ab5e57ce724173b713368977683bcf6" 2025-12-04T08:59:14.7571419Z }, 2025-12-04T08:59:14.7571608Z { 2025-12-04T08:59:14.7571958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7572405Z "size": 164, 2025-12-04T08:59:14.7572830Z "digest": "sha256:1283cd8f801a142172f3ab76fd472df8583223d9437de3e4d18d8cf98ea3fa98" 2025-12-04T08:59:14.7573339Z }, 2025-12-04T08:59:14.7573540Z { 2025-12-04T08:59:14.7573875Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7574325Z "size": 346, 2025-12-04T08:59:14.7574762Z "digest": "sha256:024fa855425fa524ad4500660cf61d53be62b99556d31b8b280d14caba434a35" 2025-12-04T08:59:14.7575255Z }, 2025-12-04T08:59:14.7575459Z { 2025-12-04T08:59:14.7575812Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7576247Z "size": 32, 2025-12-04T08:59:14.7576697Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7577220Z }, 2025-12-04T08:59:14.7577427Z { 2025-12-04T08:59:14.7577766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7578216Z "size": 106, 2025-12-04T08:59:14.7578662Z "digest": "sha256:303e6747a62efecf5efa1f97d0e66b40a3b39da8d79a51f75b89f4c92ae7ec52" 2025-12-04T08:59:14.7579167Z }, 2025-12-04T08:59:14.7579486Z { 2025-12-04T08:59:14.7579815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7580226Z "size": 424, 2025-12-04T08:59:14.7580670Z "digest": "sha256:3017cdf4838bcc9a33daebc07487f8ae1f6bd6e7ce8322c14f5480e8db9ef90e" 2025-12-04T08:59:14.7581157Z }, 2025-12-04T08:59:14.7581340Z { 2025-12-04T08:59:14.7581671Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7582105Z "size": 19309374, 2025-12-04T08:59:14.7582532Z "digest": "sha256:6b6cd1c358e886dc6ed7fd46ac4bcc1a0a73b7b1301739ea1953478ee5d83f50" 2025-12-04T08:59:14.7583031Z }, 2025-12-04T08:59:14.7583291Z { 2025-12-04T08:59:14.7583633Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7584052Z "size": 108, 2025-12-04T08:59:14.7584475Z "digest": "sha256:b2dd045011241d1cf8889e2a7369d9fe4844dfe15529b520ccd6a59bd3c1532e" 2025-12-04T08:59:14.7584961Z }, 2025-12-04T08:59:14.7585143Z { 2025-12-04T08:59:14.7585480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7585909Z "size": 827, 2025-12-04T08:59:14.7586315Z "digest": "sha256:55adc51fe5897031d4cf2f2b8fd162213f6e46a52848630c616606271b97952e" 2025-12-04T08:59:14.7586801Z }, 2025-12-04T08:59:14.7586996Z { 2025-12-04T08:59:14.7587390Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7588019Z "size": 724, 2025-12-04T08:59:14.7588524Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T08:59:14.7589063Z }, 2025-12-04T08:59:14.7589266Z { 2025-12-04T08:59:14.7589643Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7590130Z "size": 149, 2025-12-04T08:59:14.7590577Z "digest": "sha256:a43ca0e4b837964b12b7469194cfe939c26de027298040028975324dce25938a" 2025-12-04T08:59:14.7591112Z }, 2025-12-04T08:59:14.7591323Z { 2025-12-04T08:59:14.7591679Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7592152Z "size": 138, 2025-12-04T08:59:14.7592613Z "digest": "sha256:b7212f17fd1404837fcfdd086dd0e2667931e4db377d45d8d89a44390c84e11d" 2025-12-04T08:59:14.7593140Z }, 2025-12-04T08:59:14.7593351Z { 2025-12-04T08:59:14.7593720Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7594185Z "size": 141, 2025-12-04T08:59:14.7594653Z "digest": "sha256:083e42cac090e6486c35f392b64ee54448f5e4aa947003aeb3e1f92c8ea5c099" 2025-12-04T08:59:14.7595202Z }, 2025-12-04T08:59:14.7595419Z { 2025-12-04T08:59:14.7595781Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7596356Z "size": 32, 2025-12-04T08:59:14.7596831Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7597371Z }, 2025-12-04T08:59:14.7597586Z { 2025-12-04T08:59:14.7597952Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7598414Z "size": 223, 2025-12-04T08:59:14.7598882Z "digest": "sha256:0a00b784a4aac341795729b254f7edd09e811b7f51d0c58e0e6bfeeee6940503" 2025-12-04T08:59:14.7599425Z }, 2025-12-04T08:59:14.7599625Z { 2025-12-04T08:59:14.7600098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7600520Z "size": 255, 2025-12-04T08:59:14.7600923Z "digest": "sha256:c6173c779f7ba143a21214ea5f032b141863a37ceb4c0ac01d3248c216ce5241" 2025-12-04T08:59:14.7601407Z }, 2025-12-04T08:59:14.7601601Z { 2025-12-04T08:59:14.7601934Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7602354Z "size": 145520672, 2025-12-04T08:59:14.7602792Z "digest": "sha256:ed3d1e3387b924585c332bf1bc252fa159cd0d25256a874043ff0141b1ab5ff7" 2025-12-04T08:59:14.7603273Z }, 2025-12-04T08:59:14.7603455Z { 2025-12-04T08:59:14.7603788Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7604214Z "size": 106, 2025-12-04T08:59:14.7604611Z "digest": "sha256:b29343478586aeee19d2a622661716f6f1591280c890f49b727a8da13a610784" 2025-12-04T08:59:14.7605089Z }, 2025-12-04T08:59:14.7605282Z { 2025-12-04T08:59:14.7605601Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7606029Z "size": 312293530, 2025-12-04T08:59:14.7606466Z "digest": "sha256:c6f0520487fb506bc4601fd84d5f28d8a76b203e004731e4b2067c2ab1a14e0b" 2025-12-04T08:59:14.7606947Z }, 2025-12-04T08:59:14.7607128Z { 2025-12-04T08:59:14.7607459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7607887Z "size": 3058011133, 2025-12-04T08:59:14.7608375Z "digest": "sha256:148171691cd4c4d20310d490d4b4dd903490d04ea07fb8f7e668a28768683e9a" 2025-12-04T08:59:14.7608859Z }, 2025-12-04T08:59:14.7609051Z { 2025-12-04T08:59:14.7609370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7609798Z "size": 129, 2025-12-04T08:59:14.7610225Z "digest": "sha256:2c666d30ed77fff9ff1167d41cd645dad98280fcbe941f5bc3828c7ae66b1287" 2025-12-04T08:59:14.7610700Z }, 2025-12-04T08:59:14.7610894Z { 2025-12-04T08:59:14.7611223Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7611636Z "size": 880, 2025-12-04T08:59:14.7612054Z "digest": "sha256:5d8d3a0a98e012c5068e0f3bae5a03e3148ecf2d063634eee4c9241a1e3fdfb5" 2025-12-04T08:59:14.7612538Z }, 2025-12-04T08:59:14.7612733Z { 2025-12-04T08:59:14.7613052Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7613481Z "size": 724, 2025-12-04T08:59:14.7613897Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T08:59:14.7614369Z }, 2025-12-04T08:59:14.7614568Z { 2025-12-04T08:59:14.7614899Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7615309Z "size": 139, 2025-12-04T08:59:14.7615720Z "digest": "sha256:b06bafce9e817295d8127207747c80aa18e04392ff0875844fc30a1e794a8a0c" 2025-12-04T08:59:14.7616198Z }, 2025-12-04T08:59:14.7616378Z { 2025-12-04T08:59:14.7616709Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7617131Z "size": 32, 2025-12-04T08:59:14.7617540Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7618027Z }, 2025-12-04T08:59:14.7618227Z { 2025-12-04T08:59:14.7618560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7618975Z "size": 159, 2025-12-04T08:59:14.7619404Z "digest": "sha256:15e0d7e4590d3d8f598d05aec3a92f891bf8b4605bcc38cc2de852b6014ef8f3" 2025-12-04T08:59:14.7619965Z }, 2025-12-04T08:59:14.7620148Z { 2025-12-04T08:59:14.7620484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7620914Z "size": 1011, 2025-12-04T08:59:14.7621326Z "digest": "sha256:a514bd1add3164d8d7ca99aa19294c4ed8b97b074635d98714c4f598a959f4cd" 2025-12-04T08:59:14.7621820Z }, 2025-12-04T08:59:14.7622018Z { 2025-12-04T08:59:14.7622341Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7622772Z "size": 724, 2025-12-04T08:59:14.7623187Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T08:59:14.7623838Z }, 2025-12-04T08:59:14.7624200Z { 2025-12-04T08:59:14.7624574Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7625055Z "size": 134, 2025-12-04T08:59:14.7625513Z "digest": "sha256:57b84ee6000204f27a1d9bca199b19be4c86ecd324540dbdf239c56a6c3b34ea" 2025-12-04T08:59:14.7626067Z }, 2025-12-04T08:59:14.7626286Z { 2025-12-04T08:59:14.7626650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7627126Z "size": 32, 2025-12-04T08:59:14.7627682Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7628215Z }, 2025-12-04T08:59:14.7628429Z { 2025-12-04T08:59:14.7628798Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7629259Z "size": 157, 2025-12-04T08:59:14.7629738Z "digest": "sha256:b8babeff6d817a5961dddc15c6bdfdbd05da187fae75d5804015f99fd7c066d8" 2025-12-04T08:59:14.7630298Z }, 2025-12-04T08:59:14.7630520Z { 2025-12-04T08:59:14.7630879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7631359Z "size": 602, 2025-12-04T08:59:14.7631840Z "digest": "sha256:83779ddf6a85ab387f64a45f274cba245b69e4fd1931ff0b5d7d3efd4b7a43bc" 2025-12-04T08:59:14.7632373Z }, 2025-12-04T08:59:14.7632758Z { 2025-12-04T08:59:14.7633253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7633724Z "size": 724, 2025-12-04T08:59:14.7634182Z "digest": "sha256:3541df015cdb7e8925273399d28e56c31b3c9196f00439ac2925537b173b1f84" 2025-12-04T08:59:14.7634717Z }, 2025-12-04T08:59:14.7634915Z { 2025-12-04T08:59:14.7635283Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7635759Z "size": 155, 2025-12-04T08:59:14.7636212Z "digest": "sha256:8b7620c0d736cc79381207ce5afe2af90f0cd7f0cd394577d2c9520d7f74762f" 2025-12-04T08:59:14.7636755Z }, 2025-12-04T08:59:14.7636966Z { 2025-12-04T08:59:14.7637332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7637797Z "size": 32, 2025-12-04T08:59:14.7638265Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7638812Z }, 2025-12-04T08:59:14.7639121Z { 2025-12-04T08:59:14.7639587Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7640018Z "size": 188, 2025-12-04T08:59:14.7640428Z "digest": "sha256:3bcfa090e4efd3677425f76baea9f1e0c50a75d8c6b5713ec05310f1dff24539" 2025-12-04T08:59:14.7640918Z }, 2025-12-04T08:59:14.7641111Z { 2025-12-04T08:59:14.7641428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7641852Z "size": 1370, 2025-12-04T08:59:14.7642276Z "digest": "sha256:eb0504ec4d9218a79896b604f73dc0ea5a0f96266ad9c2cdbbbe5f0f18222694" 2025-12-04T08:59:14.7642764Z }, 2025-12-04T08:59:14.7642941Z { 2025-12-04T08:59:14.7643271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7643693Z "size": 32, 2025-12-04T08:59:14.7644100Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7644589Z }, 2025-12-04T08:59:14.7644782Z { 2025-12-04T08:59:14.7645100Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7645530Z "size": 136, 2025-12-04T08:59:14.7646038Z "digest": "sha256:15d0fec09d7b196a1462d51516ee90fc3443ba178d3e56d59cacf32146b4321d" 2025-12-04T08:59:14.7646510Z }, 2025-12-04T08:59:14.7646704Z { 2025-12-04T08:59:14.7647038Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7647449Z "size": 528, 2025-12-04T08:59:14.7647877Z "digest": "sha256:cca81fcc62a949959ca4dd3c9056fb293d548ef8607127eeeef6cfd3a8897ca8" 2025-12-04T08:59:14.7648367Z }, 2025-12-04T08:59:14.7648560Z { 2025-12-04T08:59:14.7648878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7649302Z "size": 32, 2025-12-04T08:59:14.7649718Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7650193Z }, 2025-12-04T08:59:14.7650390Z { 2025-12-04T08:59:14.7650721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7651132Z "size": 104, 2025-12-04T08:59:14.7651568Z "digest": "sha256:b0b8f9b5c6ab98db9cd830dc584e1b6aec9add139e4cc48d8c243d36691e25b4" 2025-12-04T08:59:14.7652069Z }, 2025-12-04T08:59:14.7652251Z { 2025-12-04T08:59:14.7652580Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7653002Z "size": 435, 2025-12-04T08:59:14.7653403Z "digest": "sha256:0606ca4d47a8a70e91e92b03ca51a85e731641b09342136a54ef2f2a6d9dfb44" 2025-12-04T08:59:14.7653883Z }, 2025-12-04T08:59:14.7654076Z { 2025-12-04T08:59:14.7654403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7654816Z "size": 32, 2025-12-04T08:59:14.7655233Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7655719Z }, 2025-12-04T08:59:14.7655899Z { 2025-12-04T08:59:14.7656235Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7656663Z "size": 109, 2025-12-04T08:59:14.7657132Z "digest": "sha256:2f80a4e1b3b95ed67bb781ea787e8a63e46de79117d9d8e65c257072b38afa2d" 2025-12-04T08:59:14.7657626Z }, 2025-12-04T08:59:14.7657823Z { 2025-12-04T08:59:14.7658144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7658575Z "size": 1896, 2025-12-04T08:59:14.7659004Z "digest": "sha256:35c916fb1bd057e517dcab78c3a2a018e68096d8993892ad84f47562d37ae352" 2025-12-04T08:59:14.7659489Z }, 2025-12-04T08:59:14.7659672Z { 2025-12-04T08:59:14.7660008Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7660437Z "size": 197526165, 2025-12-04T08:59:14.7660857Z "digest": "sha256:195537b7dafc96192f768323b1a8cc2a914d41959849b73198579576b0872a44" 2025-12-04T08:59:14.7661334Z }, 2025-12-04T08:59:14.7661530Z { 2025-12-04T08:59:14.7661849Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7662278Z "size": 106, 2025-12-04T08:59:14.7662700Z "digest": "sha256:dc454fd3967e5735b2498b7f1d958a2c626987d5e4ce225ca98da3cd945b59f3" 2025-12-04T08:59:14.7663175Z }, 2025-12-04T08:59:14.7663373Z { 2025-12-04T08:59:14.7663706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7664116Z "size": 165, 2025-12-04T08:59:14.7664535Z "digest": "sha256:701b34f115fa897181c046dc37288e87cbc3ad74c36a9e2224b5bfe7c5703afb" 2025-12-04T08:59:14.7665015Z }, 2025-12-04T08:59:14.7665207Z { 2025-12-04T08:59:14.7665524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7665947Z "size": 7944, 2025-12-04T08:59:14.7666365Z "digest": "sha256:39cefc00ffedebc9098261c798408b87a20c95a88fccb110594077f48dadf760" 2025-12-04T08:59:14.7666837Z }, 2025-12-04T08:59:14.7667030Z { 2025-12-04T08:59:14.7667441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7668056Z "size": 8071, 2025-12-04T08:59:14.7668527Z "digest": "sha256:6ae51eb61a325b2c2995a5088c81aa20821b75be65b5aa722c7c40556b5d03ea" 2025-12-04T08:59:14.7669073Z }, 2025-12-04T08:59:14.7669276Z { 2025-12-04T08:59:14.7669728Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7670204Z "size": 304, 2025-12-04T08:59:14.7670663Z "digest": "sha256:1fd5341e66dfc0c1ae23af014641a92a6fd02640c528fe6d4dc55921ed659a26" 2025-12-04T08:59:14.7671214Z }, 2025-12-04T08:59:14.7671429Z { 2025-12-04T08:59:14.7671800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7672266Z "size": 13364291, 2025-12-04T08:59:14.7672753Z "digest": "sha256:72a7c87e35e40ab796f90aee1b51add7902f0cdc44406d2505b6c6a1f55a8da6" 2025-12-04T08:59:14.7673299Z }, 2025-12-04T08:59:14.7673499Z { 2025-12-04T08:59:14.7673867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7674342Z "size": 108, 2025-12-04T08:59:14.7674812Z "digest": "sha256:ec36862ac98ebaac52ee1a8b1d162d45bd0e3bf59ae7e19c8f80ad3960b4c600" 2025-12-04T08:59:14.7675367Z }, 2025-12-04T08:59:14.7675586Z { 2025-12-04T08:59:14.7675944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7676426Z "size": 54145699, 2025-12-04T08:59:14.7676920Z "digest": "sha256:05ddbf246e8add0e293474dbf88bb028d5a295a25ac59e8648a18db644377773" 2025-12-04T08:59:14.7677468Z }, 2025-12-04T08:59:14.7677671Z { 2025-12-04T08:59:14.7678040Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-12-04T08:59:14.7678516Z "size": 32, 2025-12-04T08:59:14.7678972Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-12-04T08:59:14.7679626Z } 2025-12-04T08:59:14.7679836Z ] 2025-12-04T08:59:14.7680146Z } 2025-12-04T08:59:14.7706038Z ##[group]Run set -eux 2025-12-04T08:59:14.7706327Z set -eux 2025-12-04T08:59:14.7706749Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have 2025-12-04T08:59:14.7708385Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true 2025-12-04T08:59:14.7715673Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:14.7716112Z env: 2025-12-04T08:59:14.7716349Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:14.7716651Z ##[endgroup] 2025-12-04T08:59:14.7746688Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-12-04T08:59:14.7747379Z + jq --raw-output .SecretString 2025-12-04T08:59:14.7747733Z + jq -r .docker_hub_readonly_token 2025-12-04T08:59:14.7749286Z + docker login --username pytorchbot --password-stdin 2025-12-04T08:59:15.3555020Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:59:15.3555747Z Configure a credential helper to remove this warning. See 2025-12-04T08:59:15.3556417Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:59:15.3556871Z 2025-12-04T08:59:15.3557446Z Login Succeeded 2025-12-04T08:59:15.3648261Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T08:59:15.3648833Z tag=${ECR_DOCKER_IMAGE##*:} 2025-12-04T08:59:15.3649272Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-12-04T08:59:15.3655413Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:15.3655811Z env: 2025-12-04T08:59:15.3656035Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:15.3656901Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:15.3657786Z ##[endgroup] 2025-12-04T08:59:15.3684960Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:15.3734768Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-12-04T08:59:15.3735261Z with: 2025-12-04T08:59:15.3736316Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:15.3737700Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:15.3738104Z env: 2025-12-04T08:59:15.3738309Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:15.3738579Z ##[endgroup] 2025-12-04T08:59:15.3771516Z ##[group]Run set -x 2025-12-04T08:59:15.3771796Z set -x 2025-12-04T08:59:15.3772018Z set +e 2025-12-04T08:59:15.3772247Z  2025-12-04T08:59:15.3772465Z login() { 2025-12-04T08:59:15.3772960Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-12-04T08:59:15.3773490Z } 2025-12-04T08:59:15.3773704Z  2025-12-04T08:59:15.3773965Z retry () { 2025-12-04T08:59:15.3774247Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-12-04T08:59:15.3774562Z } 2025-12-04T08:59:15.3774775Z  2025-12-04T08:59:15.3775019Z retry login "${DOCKER_REGISTRY}" 2025-12-04T08:59:15.3775325Z  2025-12-04T08:59:15.3775834Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-12-04T08:59:15.3776524Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-12-04T08:59:15.3776908Z  2025-12-04T08:59:15.3777108Z set -e 2025-12-04T08:59:15.3777462Z # ignore output since only exit code is used for conditional 2025-12-04T08:59:15.3777973Z # only pull docker image if it's not available locally 2025-12-04T08:59:15.3778526Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-12-04T08:59:15.3779051Z  retry docker pull "${DOCKER_IMAGE}" 2025-12-04T08:59:15.3779381Z fi 2025-12-04T08:59:15.3784792Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T08:59:15.3785185Z env: 2025-12-04T08:59:15.3785406Z GIT_DEFAULT_BRANCH: main 2025-12-04T08:59:15.3786274Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:15.3787351Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:15.3787950Z ##[endgroup] 2025-12-04T08:59:15.3812651Z + set +e 2025-12-04T08:59:15.3813135Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:15.3813619Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:15.3817069Z + aws ecr get-login-password --region us-east-1 2025-12-04T08:59:15.3818125Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-12-04T08:59:15.9015064Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-12-04T08:59:15.9015768Z Configure a credential helper to remove this warning. See 2025-12-04T08:59:15.9016416Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-12-04T08:59:15.9016886Z 2025-12-04T08:59:15.9017488Z Login Succeeded 2025-12-04T08:59:15.9037777Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:15.9039021Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-12-04T08:59:16.1054772Z + IMAGE_SIZE=15091.581844329834 2025-12-04T08:59:16.1055208Z Compressed size of image in MB: 15091.581844329834 2025-12-04T08:59:16.1055821Z + echo 'Compressed size of image in MB: 15091.581844329834' 2025-12-04T08:59:16.1056257Z + set -e 2025-12-04T08:59:16.1057481Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:16.1177023Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:16.1178876Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T08:59:16.3482663Z pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a: Pulling from pytorch/ci-image 2025-12-04T08:59:16.3484411Z 63e5bc7682b8: Pulling fs layer 2025-12-04T08:59:16.3484724Z 0678d56345c9: Pulling fs layer 2025-12-04T08:59:16.3485044Z 45f5c9ddfce7: Pulling fs layer 2025-12-04T08:59:16.3485361Z 086b1df51ac1: Pulling fs layer 2025-12-04T08:59:16.3485790Z fe8a7b64bf98: Pulling fs layer 2025-12-04T08:59:16.3486122Z 7680723e9a57: Pulling fs layer 2025-12-04T08:59:16.3486456Z 9c5027aeeb4e: Pulling fs layer 2025-12-04T08:59:16.3486758Z 9a5652110360: Pulling fs layer 2025-12-04T08:59:16.3487074Z 375c4427e914: Pulling fs layer 2025-12-04T08:59:16.3487390Z a86faaa7dbdd: Pulling fs layer 2025-12-04T08:59:16.3487711Z fb7848686804: Pulling fs layer 2025-12-04T08:59:16.3488019Z 3541df015cdb: Pulling fs layer 2025-12-04T08:59:16.3488331Z 79dc80f426b2: Pulling fs layer 2025-12-04T08:59:16.3488652Z a13fcc1b90bb: Pulling fs layer 2025-12-04T08:59:16.3489040Z 4f4fb700ef54: Pulling fs layer 2025-12-04T08:59:16.3489353Z 549db4d6c618: Pulling fs layer 2025-12-04T08:59:16.3489668Z 5c63528cb580: Pulling fs layer 2025-12-04T08:59:16.3489967Z 75bd83b989a4: Pulling fs layer 2025-12-04T08:59:16.3490277Z de6e78970f51: Pulling fs layer 2025-12-04T08:59:16.3490587Z e13ed7c7e473: Pulling fs layer 2025-12-04T08:59:16.3490884Z 6e2949bcb741: Pulling fs layer 2025-12-04T08:59:16.3491192Z 14d69d9aaec7: Pulling fs layer 2025-12-04T08:59:16.3491501Z 5c02769dd8e5: Pulling fs layer 2025-12-04T08:59:16.3491837Z 7680723e9a57: Waiting 2025-12-04T08:59:16.3492094Z 9a5652110360: Waiting 2025-12-04T08:59:16.3492371Z 35041ce524ac: Pulling fs layer 2025-12-04T08:59:16.3492677Z 9c5027aeeb4e: Waiting 2025-12-04T08:59:16.3492931Z 375c4427e914: Waiting 2025-12-04T08:59:16.3493218Z 2fa92dc5885e: Pulling fs layer 2025-12-04T08:59:16.3493535Z 2b85eafbd92a: Pulling fs layer 2025-12-04T08:59:16.3493855Z ff755a4ddad7: Pulling fs layer 2025-12-04T08:59:16.3494145Z 086b1df51ac1: Waiting 2025-12-04T08:59:16.3494414Z a86faaa7dbdd: Waiting 2025-12-04T08:59:16.3494696Z 09eb41bdf42d: Pulling fs layer 2025-12-04T08:59:16.3494984Z 79dc80f426b2: Waiting 2025-12-04T08:59:16.3495261Z 11ede4d59e93: Pulling fs layer 2025-12-04T08:59:16.3495561Z 4f4fb700ef54: Waiting 2025-12-04T08:59:16.3495810Z fb7848686804: Waiting 2025-12-04T08:59:16.3496073Z a13fcc1b90bb: Waiting 2025-12-04T08:59:16.3496349Z 1283cd8f801a: Pulling fs layer 2025-12-04T08:59:16.3496639Z 549db4d6c618: Waiting 2025-12-04T08:59:16.3496915Z 024fa855425f: Pulling fs layer 2025-12-04T08:59:16.3497221Z 3541df015cdb: Waiting 2025-12-04T08:59:16.3497470Z 5c63528cb580: Waiting 2025-12-04T08:59:16.3497730Z e13ed7c7e473: Waiting 2025-12-04T08:59:16.3498099Z 75bd83b989a4: Waiting 2025-12-04T08:59:16.3498379Z de6e78970f51: Waiting 2025-12-04T08:59:16.3498661Z 303e6747a62e: Pulling fs layer 2025-12-04T08:59:16.3498973Z 3017cdf4838b: Pulling fs layer 2025-12-04T08:59:16.3499447Z fe8a7b64bf98: Waiting 2025-12-04T08:59:16.3499733Z 6b6cd1c358e8: Pulling fs layer 2025-12-04T08:59:16.3500042Z 6e2949bcb741: Waiting 2025-12-04T08:59:16.3500311Z b2dd04501124: Pulling fs layer 2025-12-04T08:59:16.3500618Z 14d69d9aaec7: Waiting 2025-12-04T08:59:16.3501019Z 5c02769dd8e5: Waiting 2025-12-04T08:59:16.3501286Z 2b85eafbd92a: Waiting 2025-12-04T08:59:16.3501556Z 2fa92dc5885e: Waiting 2025-12-04T08:59:16.3501823Z 55adc51fe589: Pulling fs layer 2025-12-04T08:59:16.3502130Z 35041ce524ac: Waiting 2025-12-04T08:59:16.3502408Z a43ca0e4b837: Pulling fs layer 2025-12-04T08:59:16.3502711Z b7212f17fd14: Pulling fs layer 2025-12-04T08:59:16.3503234Z ff755a4ddad7: Waiting 2025-12-04T08:59:16.3503508Z 1283cd8f801a: Waiting 2025-12-04T08:59:16.3503774Z 083e42cac090: Pulling fs layer 2025-12-04T08:59:16.3504077Z 3017cdf4838b: Waiting 2025-12-04T08:59:16.3504345Z 024fa855425f: Waiting 2025-12-04T08:59:16.3504693Z 303e6747a62e: Waiting 2025-12-04T08:59:16.3504960Z 6b6cd1c358e8: Waiting 2025-12-04T08:59:16.3505228Z 11ede4d59e93: Waiting 2025-12-04T08:59:16.3505540Z 0a00b784a4aa: Pulling fs layer 2025-12-04T08:59:16.3505872Z 09eb41bdf42d: Waiting 2025-12-04T08:59:16.3506153Z c6173c779f7b: Pulling fs layer 2025-12-04T08:59:16.3506441Z 55adc51fe589: Waiting 2025-12-04T08:59:16.3506709Z b2dd04501124: Waiting 2025-12-04T08:59:16.3507051Z a43ca0e4b837: Waiting 2025-12-04T08:59:16.3507405Z ed3d1e3387b9: Pulling fs layer 2025-12-04T08:59:16.3507894Z b7212f17fd14: Waiting 2025-12-04T08:59:16.3508167Z 0a00b784a4aa: Waiting 2025-12-04T08:59:16.3508446Z b29343478586: Pulling fs layer 2025-12-04T08:59:16.3508740Z 083e42cac090: Waiting 2025-12-04T08:59:16.3509019Z c6173c779f7b: Waiting 2025-12-04T08:59:16.3509303Z c6f0520487fb: Pulling fs layer 2025-12-04T08:59:16.3509612Z 148171691cd4: Pulling fs layer 2025-12-04T08:59:16.3509921Z ed3d1e3387b9: Waiting 2025-12-04T08:59:16.3510192Z b29343478586: Waiting 2025-12-04T08:59:16.3510468Z 2c666d30ed77: Pulling fs layer 2025-12-04T08:59:16.3510777Z c6f0520487fb: Waiting 2025-12-04T08:59:16.3511058Z 5d8d3a0a98e0: Pulling fs layer 2025-12-04T08:59:16.3511369Z b06bafce9e81: Pulling fs layer 2025-12-04T08:59:16.3511693Z 15e0d7e4590d: Pulling fs layer 2025-12-04T08:59:16.3512019Z a514bd1add31: Pulling fs layer 2025-12-04T08:59:16.3512317Z b06bafce9e81: Waiting 2025-12-04T08:59:16.3512606Z 57b84ee60002: Pulling fs layer 2025-12-04T08:59:16.3512913Z 15e0d7e4590d: Waiting 2025-12-04T08:59:16.3513168Z 2c666d30ed77: Waiting 2025-12-04T08:59:16.3513486Z 148171691cd4: Waiting 2025-12-04T08:59:16.3513753Z 57b84ee60002: Waiting 2025-12-04T08:59:16.3514027Z b8babeff6d81: Pulling fs layer 2025-12-04T08:59:16.3514343Z 5d8d3a0a98e0: Waiting 2025-12-04T08:59:16.3514674Z 83779ddf6a85: Pulling fs layer 2025-12-04T08:59:16.3514992Z 8b7620c0d736: Pulling fs layer 2025-12-04T08:59:16.3515293Z b8babeff6d81: Waiting 2025-12-04T08:59:16.3515583Z 3bcfa090e4ef: Pulling fs layer 2025-12-04T08:59:16.3515958Z eb0504ec4d92: Pulling fs layer 2025-12-04T08:59:16.3516271Z 15d0fec09d7b: Pulling fs layer 2025-12-04T08:59:16.3516624Z cca81fcc62a9: Pulling fs layer 2025-12-04T08:59:16.3516950Z b0b8f9b5c6ab: Pulling fs layer 2025-12-04T08:59:16.3517326Z 83779ddf6a85: Waiting 2025-12-04T08:59:16.3517606Z 3bcfa090e4ef: Waiting 2025-12-04T08:59:16.3517888Z cca81fcc62a9: Waiting 2025-12-04T08:59:16.3518168Z 15d0fec09d7b: Waiting 2025-12-04T08:59:16.3518441Z 0606ca4d47a8: Pulling fs layer 2025-12-04T08:59:16.3518747Z 8b7620c0d736: Waiting 2025-12-04T08:59:16.3519017Z eb0504ec4d92: Waiting 2025-12-04T08:59:16.3519275Z b0b8f9b5c6ab: Waiting 2025-12-04T08:59:16.3519549Z 0606ca4d47a8: Waiting 2025-12-04T08:59:16.3519936Z 2f80a4e1b3b9: Pulling fs layer 2025-12-04T08:59:16.3520229Z 2f80a4e1b3b9: Waiting 2025-12-04T08:59:16.3520504Z 35c916fb1bd0: Pulling fs layer 2025-12-04T08:59:16.3520817Z 195537b7dafc: Pulling fs layer 2025-12-04T08:59:16.3521115Z dc454fd3967e: Pulling fs layer 2025-12-04T08:59:16.3521430Z 701b34f115fa: Pulling fs layer 2025-12-04T08:59:16.3521747Z 39cefc00ffed: Pulling fs layer 2025-12-04T08:59:16.3522048Z 6ae51eb61a32: Pulling fs layer 2025-12-04T08:59:16.3522347Z dc454fd3967e: Waiting 2025-12-04T08:59:16.3522611Z 35c916fb1bd0: Waiting 2025-12-04T08:59:16.3522873Z 1fd5341e66df: Pulling fs layer 2025-12-04T08:59:16.3523174Z 701b34f115fa: Waiting 2025-12-04T08:59:16.3523480Z 195537b7dafc: Waiting 2025-12-04T08:59:16.3523902Z 39cefc00ffed: Waiting 2025-12-04T08:59:16.3524185Z 72a7c87e35e4: Pulling fs layer 2025-12-04T08:59:16.3524703Z 6ae51eb61a32: Waiting 2025-12-04T08:59:16.3524991Z ec36862ac98e: Pulling fs layer 2025-12-04T08:59:16.3525290Z 1fd5341e66df: Waiting 2025-12-04T08:59:16.3525580Z 05ddbf246e8a: Pulling fs layer 2025-12-04T08:59:16.3525892Z 72a7c87e35e4: Waiting 2025-12-04T08:59:16.3526315Z 05ddbf246e8a: Waiting 2025-12-04T08:59:16.3526597Z ec36862ac98e: Waiting 2025-12-04T08:59:16.4409773Z 0678d56345c9: Verifying Checksum 2025-12-04T08:59:16.4410220Z 0678d56345c9: Download complete 2025-12-04T08:59:16.5172560Z 086b1df51ac1: Download complete 2025-12-04T08:59:16.5941879Z fe8a7b64bf98: Verifying Checksum 2025-12-04T08:59:16.5942380Z fe8a7b64bf98: Download complete 2025-12-04T08:59:16.6864397Z 7680723e9a57: Verifying Checksum 2025-12-04T08:59:16.6864902Z 7680723e9a57: Download complete 2025-12-04T08:59:16.6951165Z 63e5bc7682b8: Verifying Checksum 2025-12-04T08:59:16.6951581Z 63e5bc7682b8: Download complete 2025-12-04T08:59:16.7533998Z 9c5027aeeb4e: Download complete 2025-12-04T08:59:16.7701766Z 9a5652110360: Verifying Checksum 2025-12-04T08:59:16.7702131Z 9a5652110360: Download complete 2025-12-04T08:59:16.8406429Z a86faaa7dbdd: Verifying Checksum 2025-12-04T08:59:16.8406829Z a86faaa7dbdd: Download complete 2025-12-04T08:59:16.9170222Z fb7848686804: Verifying Checksum 2025-12-04T08:59:16.9170660Z fb7848686804: Download complete 2025-12-04T08:59:16.9721980Z 3541df015cdb: Verifying Checksum 2025-12-04T08:59:16.9722414Z 3541df015cdb: Download complete 2025-12-04T08:59:17.0767374Z 79dc80f426b2: Verifying Checksum 2025-12-04T08:59:17.5083735Z 79dc80f426b2: Download complete 2025-12-04T08:59:17.5084149Z 63e5bc7682b8: Pull complete 2025-12-04T08:59:17.5300485Z 0678d56345c9: Pull complete 2025-12-04T08:59:17.9079361Z 375c4427e914: Verifying Checksum 2025-12-04T08:59:17.9079870Z 375c4427e914: Download complete 2025-12-04T08:59:17.9146225Z 4f4fb700ef54: Verifying Checksum 2025-12-04T08:59:17.9146734Z 4f4fb700ef54: Download complete 2025-12-04T08:59:17.9928415Z 549db4d6c618: Download complete 2025-12-04T08:59:18.1012773Z 5c63528cb580: Download complete 2025-12-04T08:59:18.1790003Z 75bd83b989a4: Verifying Checksum 2025-12-04T08:59:18.1790682Z 75bd83b989a4: Download complete 2025-12-04T08:59:18.2669926Z de6e78970f51: Verifying Checksum 2025-12-04T08:59:18.2671043Z de6e78970f51: Download complete 2025-12-04T08:59:18.3362896Z e13ed7c7e473: Verifying Checksum 2025-12-04T08:59:18.3363922Z e13ed7c7e473: Download complete 2025-12-04T08:59:18.4170301Z 6e2949bcb741: Verifying Checksum 2025-12-04T08:59:18.4170744Z 6e2949bcb741: Download complete 2025-12-04T08:59:18.4931139Z 14d69d9aaec7: Verifying Checksum 2025-12-04T08:59:18.4932760Z 14d69d9aaec7: Download complete 2025-12-04T08:59:18.5597292Z 5c02769dd8e5: Verifying Checksum 2025-12-04T08:59:18.5597909Z 5c02769dd8e5: Download complete 2025-12-04T08:59:19.5853372Z 45f5c9ddfce7: Verifying Checksum 2025-12-04T08:59:19.5853787Z 45f5c9ddfce7: Download complete 2025-12-04T08:59:19.6505937Z 2fa92dc5885e: Verifying Checksum 2025-12-04T08:59:19.6506340Z 2fa92dc5885e: Download complete 2025-12-04T08:59:20.0283537Z 2b85eafbd92a: Verifying Checksum 2025-12-04T08:59:20.0283963Z 2b85eafbd92a: Download complete 2025-12-04T08:59:20.1184357Z ff755a4ddad7: Verifying Checksum 2025-12-04T08:59:20.1184799Z ff755a4ddad7: Download complete 2025-12-04T08:59:20.1995515Z 09eb41bdf42d: Download complete 2025-12-04T08:59:24.8260315Z 11ede4d59e93: Verifying Checksum 2025-12-04T08:59:24.8260740Z 11ede4d59e93: Download complete 2025-12-04T08:59:24.8953277Z 1283cd8f801a: Verifying Checksum 2025-12-04T08:59:24.8953727Z 1283cd8f801a: Download complete 2025-12-04T08:59:24.9779718Z 024fa855425f: Verifying Checksum 2025-12-04T08:59:24.9780291Z 024fa855425f: Download complete 2025-12-04T08:59:25.0595808Z 303e6747a62e: Verifying Checksum 2025-12-04T08:59:25.0596416Z 303e6747a62e: Download complete 2025-12-04T08:59:25.1509336Z 3017cdf4838b: Verifying Checksum 2025-12-04T08:59:25.1509946Z 3017cdf4838b: Download complete 2025-12-04T08:59:25.3934805Z 6b6cd1c358e8: Verifying Checksum 2025-12-04T08:59:25.3935282Z 6b6cd1c358e8: Download complete 2025-12-04T08:59:25.4568605Z b2dd04501124: Verifying Checksum 2025-12-04T08:59:25.4569202Z b2dd04501124: Download complete 2025-12-04T08:59:25.5300333Z 55adc51fe589: Verifying Checksum 2025-12-04T08:59:25.5300911Z 55adc51fe589: Download complete 2025-12-04T08:59:25.6064423Z a43ca0e4b837: Verifying Checksum 2025-12-04T08:59:25.6065104Z a43ca0e4b837: Download complete 2025-12-04T08:59:25.6868365Z b7212f17fd14: Verifying Checksum 2025-12-04T08:59:25.6868927Z b7212f17fd14: Download complete 2025-12-04T08:59:25.7669838Z 083e42cac090: Verifying Checksum 2025-12-04T08:59:25.7670391Z 083e42cac090: Download complete 2025-12-04T08:59:25.8441966Z 0a00b784a4aa: Verifying Checksum 2025-12-04T08:59:25.8442539Z 0a00b784a4aa: Download complete 2025-12-04T08:59:25.9176088Z c6173c779f7b: Verifying Checksum 2025-12-04T08:59:25.9176690Z c6173c779f7b: Download complete 2025-12-04T08:59:26.4497021Z 45f5c9ddfce7: Pull complete 2025-12-04T08:59:26.4709709Z 086b1df51ac1: Pull complete 2025-12-04T08:59:26.4928832Z fe8a7b64bf98: Pull complete 2025-12-04T08:59:26.5139825Z 7680723e9a57: Pull complete 2025-12-04T08:59:26.5358137Z 9c5027aeeb4e: Pull complete 2025-12-04T08:59:26.5573877Z 9a5652110360: Pull complete 2025-12-04T08:59:27.4202675Z ed3d1e3387b9: Verifying Checksum 2025-12-04T08:59:27.4203136Z ed3d1e3387b9: Download complete 2025-12-04T08:59:27.4905095Z b29343478586: Verifying Checksum 2025-12-04T08:59:27.4905533Z b29343478586: Download complete 2025-12-04T08:59:28.8735953Z 375c4427e914: Pull complete 2025-12-04T08:59:29.1791390Z a86faaa7dbdd: Pull complete 2025-12-04T08:59:29.5789734Z fb7848686804: Pull complete 2025-12-04T08:59:30.0005785Z 3541df015cdb: Pull complete 2025-12-04T08:59:30.5033745Z 79dc80f426b2: Pull complete 2025-12-04T08:59:30.6679386Z c6f0520487fb: Verifying Checksum 2025-12-04T08:59:30.6679946Z c6f0520487fb: Download complete 2025-12-04T08:59:48.9909497Z a13fcc1b90bb: Verifying Checksum 2025-12-04T08:59:48.9909954Z a13fcc1b90bb: Download complete 2025-12-04T08:59:49.0626027Z 2c666d30ed77: Download complete 2025-12-04T08:59:49.1614286Z 5d8d3a0a98e0: Download complete 2025-12-04T08:59:49.2333569Z b06bafce9e81: Verifying Checksum 2025-12-04T08:59:49.2333999Z b06bafce9e81: Download complete 2025-12-04T08:59:49.3432178Z 15e0d7e4590d: Verifying Checksum 2025-12-04T08:59:49.3432618Z 15e0d7e4590d: Download complete 2025-12-04T08:59:49.3990152Z a514bd1add31: Verifying Checksum 2025-12-04T08:59:49.3990543Z a514bd1add31: Download complete 2025-12-04T08:59:49.4725671Z 57b84ee60002: Verifying Checksum 2025-12-04T08:59:49.4726101Z 57b84ee60002: Download complete 2025-12-04T08:59:49.5475468Z b8babeff6d81: Verifying Checksum 2025-12-04T08:59:49.5475869Z b8babeff6d81: Download complete 2025-12-04T08:59:49.6115796Z 83779ddf6a85: Verifying Checksum 2025-12-04T08:59:49.6116202Z 83779ddf6a85: Download complete 2025-12-04T08:59:49.6751878Z 8b7620c0d736: Verifying Checksum 2025-12-04T08:59:49.6752282Z 8b7620c0d736: Download complete 2025-12-04T08:59:49.7555038Z 3bcfa090e4ef: Verifying Checksum 2025-12-04T08:59:49.7555459Z 3bcfa090e4ef: Download complete 2025-12-04T08:59:49.8271211Z eb0504ec4d92: Verifying Checksum 2025-12-04T08:59:49.8271654Z eb0504ec4d92: Download complete 2025-12-04T08:59:49.9055180Z 15d0fec09d7b: Verifying Checksum 2025-12-04T08:59:49.9055633Z 15d0fec09d7b: Download complete 2025-12-04T08:59:49.9877346Z cca81fcc62a9: Verifying Checksum 2025-12-04T08:59:49.9877765Z cca81fcc62a9: Download complete 2025-12-04T08:59:50.0544841Z b0b8f9b5c6ab: Verifying Checksum 2025-12-04T08:59:50.0545292Z b0b8f9b5c6ab: Download complete 2025-12-04T08:59:50.1211584Z 0606ca4d47a8: Verifying Checksum 2025-12-04T08:59:50.1212092Z 0606ca4d47a8: Download complete 2025-12-04T08:59:50.1910952Z 2f80a4e1b3b9: Verifying Checksum 2025-12-04T08:59:50.1911387Z 2f80a4e1b3b9: Download complete 2025-12-04T08:59:50.3119460Z 35c916fb1bd0: Verifying Checksum 2025-12-04T08:59:50.3119983Z 35c916fb1bd0: Download complete 2025-12-04T08:59:52.3456831Z 195537b7dafc: Verifying Checksum 2025-12-04T08:59:52.3457265Z 195537b7dafc: Download complete 2025-12-04T08:59:52.4179186Z dc454fd3967e: Verifying Checksum 2025-12-04T08:59:52.4179611Z dc454fd3967e: Download complete 2025-12-04T08:59:52.4918007Z 701b34f115fa: Download complete 2025-12-04T08:59:52.5599609Z 39cefc00ffed: Download complete 2025-12-04T08:59:52.6360970Z 6ae51eb61a32: Download complete 2025-12-04T08:59:52.7031971Z 1fd5341e66df: Verifying Checksum 2025-12-04T08:59:52.7032389Z 1fd5341e66df: Download complete 2025-12-04T08:59:52.8804227Z 72a7c87e35e4: Verifying Checksum 2025-12-04T08:59:52.8805009Z 72a7c87e35e4: Download complete 2025-12-04T08:59:52.9442124Z ec36862ac98e: Verifying Checksum 2025-12-04T08:59:52.9442547Z ec36862ac98e: Download complete 2025-12-04T08:59:54.8826148Z 05ddbf246e8a: Verifying Checksum 2025-12-04T08:59:54.8826608Z 05ddbf246e8a: Download complete 2025-12-04T09:00:01.2883399Z 148171691cd4: Verifying Checksum 2025-12-04T09:00:01.2883798Z 148171691cd4: Download complete 2025-12-04T09:00:33.7775570Z a13fcc1b90bb: Pull complete 2025-12-04T09:00:34.0429786Z 4f4fb700ef54: Pull complete 2025-12-04T09:00:34.3056773Z 549db4d6c618: Pull complete 2025-12-04T09:00:34.7344766Z 5c63528cb580: Pull complete 2025-12-04T09:00:35.1428977Z 75bd83b989a4: Pull complete 2025-12-04T09:00:35.6233913Z de6e78970f51: Pull complete 2025-12-04T09:00:36.1546220Z e13ed7c7e473: Pull complete 2025-12-04T09:00:36.5686061Z 6e2949bcb741: Pull complete 2025-12-04T09:00:36.9530840Z 14d69d9aaec7: Pull complete 2025-12-04T09:00:37.4345303Z 5c02769dd8e5: Pull complete 2025-12-04T09:00:37.5009242Z 35041ce524ac: Verifying Checksum 2025-12-04T09:00:37.5009650Z 35041ce524ac: Download complete 2025-12-04T09:01:50.0019640Z 35041ce524ac: Pull complete 2025-12-04T09:01:50.4509257Z 2fa92dc5885e: Pull complete 2025-12-04T09:01:51.4728391Z 2b85eafbd92a: Pull complete 2025-12-04T09:01:51.9200692Z ff755a4ddad7: Pull complete 2025-12-04T09:01:52.2708866Z 09eb41bdf42d: Pull complete 2025-12-04T09:02:00.3253584Z 11ede4d59e93: Pull complete 2025-12-04T09:02:00.5789141Z 1283cd8f801a: Pull complete 2025-12-04T09:02:00.8847940Z 024fa855425f: Pull complete 2025-12-04T09:02:01.5913813Z 303e6747a62e: Pull complete 2025-12-04T09:02:01.6173422Z 3017cdf4838b: Pull complete 2025-12-04T09:02:01.8354406Z 6b6cd1c358e8: Pull complete 2025-12-04T09:02:01.8594809Z b2dd04501124: Pull complete 2025-12-04T09:02:01.8836733Z 55adc51fe589: Pull complete 2025-12-04T09:02:01.9290493Z a43ca0e4b837: Pull complete 2025-12-04T09:02:01.9528147Z b7212f17fd14: Pull complete 2025-12-04T09:02:01.9776382Z 083e42cac090: Pull complete 2025-12-04T09:02:02.0243933Z 0a00b784a4aa: Pull complete 2025-12-04T09:02:02.0469756Z c6173c779f7b: Pull complete 2025-12-04T09:02:04.9640702Z ed3d1e3387b9: Pull complete 2025-12-04T09:02:04.9898311Z b29343478586: Pull complete 2025-12-04T09:02:06.3691416Z c6f0520487fb: Pull complete 2025-12-04T09:02:58.0486977Z 148171691cd4: Pull complete 2025-12-04T09:02:58.3907006Z 2c666d30ed77: Pull complete 2025-12-04T09:02:58.8430092Z 5d8d3a0a98e0: Pull complete 2025-12-04T09:02:59.7010671Z b06bafce9e81: Pull complete 2025-12-04T09:03:00.6697535Z 15e0d7e4590d: Pull complete 2025-12-04T09:03:01.1346662Z a514bd1add31: Pull complete 2025-12-04T09:03:01.7897190Z 57b84ee60002: Pull complete 2025-12-04T09:03:02.3761583Z b8babeff6d81: Pull complete 2025-12-04T09:03:02.6994134Z 83779ddf6a85: Pull complete 2025-12-04T09:03:03.3063700Z 8b7620c0d736: Pull complete 2025-12-04T09:03:03.9590464Z 3bcfa090e4ef: Pull complete 2025-12-04T09:03:04.2108978Z eb0504ec4d92: Pull complete 2025-12-04T09:03:04.8778715Z 15d0fec09d7b: Pull complete 2025-12-04T09:03:05.2938053Z cca81fcc62a9: Pull complete 2025-12-04T09:03:06.2422979Z b0b8f9b5c6ab: Pull complete 2025-12-04T09:03:06.6251569Z 0606ca4d47a8: Pull complete 2025-12-04T09:03:07.4667543Z 2f80a4e1b3b9: Pull complete 2025-12-04T09:03:07.8725677Z 35c916fb1bd0: Pull complete 2025-12-04T09:03:13.8382035Z 195537b7dafc: Pull complete 2025-12-04T09:03:14.1700300Z dc454fd3967e: Pull complete 2025-12-04T09:03:14.4909516Z 701b34f115fa: Pull complete 2025-12-04T09:03:14.8974963Z 39cefc00ffed: Pull complete 2025-12-04T09:03:15.3294068Z 6ae51eb61a32: Pull complete 2025-12-04T09:03:15.8557473Z 1fd5341e66df: Pull complete 2025-12-04T09:03:17.4209457Z 72a7c87e35e4: Pull complete 2025-12-04T09:03:17.8166777Z ec36862ac98e: Pull complete 2025-12-04T09:03:19.4712754Z 05ddbf246e8a: Pull complete 2025-12-04T09:03:20.0739951Z Digest: sha256:ba21003510dba4bdeed83df81a56fa468e0ee1b612a9445ae1f402a280804f97 2025-12-04T09:03:20.1703644Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:03:20.2155760Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:03:20.2209441Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:03:20.2210538Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:03:20.2220050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:03:20.2220469Z env: 2025-12-04T09:03:20.2220695Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:03:20.2220984Z ##[endgroup] 2025-12-04T09:03:20.2416192Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-12-04T09:03:20.2416661Z with: 2025-12-04T09:03:20.2416916Z driver-version: 580.82.07 2025-12-04T09:03:20.2417194Z env: 2025-12-04T09:03:20.2417443Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:03:20.2417738Z ##[endgroup] 2025-12-04T09:03:20.2482416Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:03:20.2483467Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-12-04T09:03:20.2490111Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:03:20.2490514Z env: 2025-12-04T09:03:20.2490740Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:03:20.2491000Z ##[endgroup] 2025-12-04T09:03:20.2666086Z ##[group]Run set -euo pipefail 2025-12-04T09:03:20.2666451Z set -euo pipefail 2025-12-04T09:03:20.2666799Z  2025-12-04T09:03:20.2667028Z has_gpu=false 2025-12-04T09:03:20.2667427Z devices="" 2025-12-04T09:03:20.2667853Z  2025-12-04T09:03:20.2668160Z if command -v nvidia-smi >/dev/null 2>&1; then 2025-12-04T09:03:20.2668698Z  if nvidia-smi -L >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:03:20.2669156Z  has_gpu=true 2025-12-04T09:03:20.2709137Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:03:20.2709605Z  fi 2025-12-04T09:03:20.2709850Z fi 2025-12-04T09:03:20.2710095Z  2025-12-04T09:03:20.2710365Z if [ "$has_gpu" = false ]; then 2025-12-04T09:03:20.2710817Z  if ls /dev/nvidia* >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:03:20.2711271Z  has_gpu=true 2025-12-04T09:03:20.2711619Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:03:20.2711980Z  fi 2025-12-04T09:03:20.2712231Z fi 2025-12-04T09:03:20.2712486Z  2025-12-04T09:03:20.2712837Z if [ "$has_gpu" = false ] && command -v lspci >/dev/null 2>&1; then 2025-12-04T09:03:20.2713445Z  if lspci | grep -i 'nvidia' >/tmp/nvidia_devices 2>/dev/null; then 2025-12-04T09:03:20.2713943Z  has_gpu=true 2025-12-04T09:03:20.2714288Z  devices=$(cat /tmp/nvidia_devices) 2025-12-04T09:03:20.2714644Z  fi 2025-12-04T09:03:20.2714891Z fi 2025-12-04T09:03:20.2715157Z  2025-12-04T09:03:20.2715497Z printf 'HAS_NVIDIA=%s\n' "$has_gpu" >> "$GITHUB_OUTPUT" 2025-12-04T09:03:20.2716135Z printf 'DETECTED_DEVICES<> "$GITHUB_OUTPUT" 2025-12-04T09:03:20.2722358Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:03:20.2722782Z env: 2025-12-04T09:03:20.2723022Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:03:20.2723300Z ##[endgroup] 2025-12-04T09:03:23.3732514Z ##[group]Run if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T09:03:23.3732990Z if [ "${HAS_NVIDIA}" = "true" ]; then 2025-12-04T09:03:23.3733420Z  echo "HAS_NVIDIA_GPU=true" >> "${GITHUB_ENV}" 2025-12-04T09:03:23.3734026Z  echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2025-12-04T09:03:23.3734734Z else 2025-12-04T09:03:23.3735053Z  echo "HAS_NVIDIA_GPU=false" >> "${GITHUB_ENV}" 2025-12-04T09:03:23.3735562Z fi 2025-12-04T09:03:23.3741566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:03:23.3741943Z env: 2025-12-04T09:03:23.3742169Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:03:23.3742438Z HAS_NVIDIA: true 2025-12-04T09:03:23.3742673Z ##[endgroup] 2025-12-04T09:03:23.3834148Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2025-12-04T09:03:23.3834637Z with: 2025-12-04T09:03:23.3834868Z timeout_minutes: 10 2025-12-04T09:03:23.3835150Z max_attempts: 3 2025-12-04T09:03:23.3865851Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y \ nvidia-container-toolkit-1.17.8 \ libnvidia-container-tools-1.17.8 \ libnvidia-container1-1.17.8 \ nvidia-container-toolkit-base-1.17.8 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-container-toolkit-1.17.8 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" # Turn off persistent mode so that the installation script can unload the kernel module sudo killall nvidia-persistenced || true else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi # check if the container-toolkit is correctly installed and CUDA is available inside a container docker run --rm -t --gpus=all public.ecr.aws/docker/library/python:3.13 nvidia-smi 2025-12-04T09:03:23.3897559Z retry_wait_seconds: 10 2025-12-04T09:03:23.3897847Z polling_interval_seconds: 1 2025-12-04T09:03:23.3898126Z warning_on_retry: true 2025-12-04T09:03:23.3898411Z continue_on_error: false 2025-12-04T09:03:23.3898677Z env: 2025-12-04T09:03:23.3898881Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:03:23.3899153Z HAS_NVIDIA_GPU: true 2025-12-04T09:03:23.3899482Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:03:23.3899854Z DRIVER_VERSION: 580.82.07 2025-12-04T09:03:23.3900122Z ##[endgroup] 2025-12-04T09:03:23.5282368Z == Installing nvidia driver NVIDIA-Linux-x86_64-580.82.07.run == 2025-12-04T09:03:23.5282929Z + pre_install_nvidia_driver_amzn2 2025-12-04T09:03:23.5284268Z + sudo yum remove -y nvidia-driver-latest-dkms 2025-12-04T09:03:24.1950335Z No match for argument: nvidia-driver-latest-dkms 2025-12-04T09:03:24.1951155Z No packages marked for removal. 2025-12-04T09:03:24.2018347Z Dependencies resolved. 2025-12-04T09:03:24.2028254Z Nothing to do. 2025-12-04T09:03:24.2029376Z Complete! 2025-12-04T09:03:24.3184828Z + install_nvidia_driver_common 2025-12-04T09:03:24.3187990Z + echo 'Before installing NVIDIA driver' 2025-12-04T09:03:24.3190272Z + lspci 2025-12-04T09:03:24.3192764Z Before installing NVIDIA driver 2025-12-04T09:03:24.4584438Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-12-04T09:03:24.4585042Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-12-04T09:03:24.4585976Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-12-04T09:03:24.4586623Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-12-04T09:03:24.4587318Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-12-04T09:03:24.4588239Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-12-04T09:03:24.4588870Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-12-04T09:03:24.4589451Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-12-04T09:03:24.4590010Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-12-04T09:03:24.4590738Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-12-04T09:03:24.4591362Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-12-04T09:03:24.4591866Z + lsmod 2025-12-04T09:03:24.4622637Z Module Size Used by 2025-12-04T09:03:24.4623130Z nvidia_uvm 1925120 0 2025-12-04T09:03:24.4623465Z nvidia 14286848 1 nvidia_uvm 2025-12-04T09:03:24.4624109Z drm 602112 1 nvidia 2025-12-04T09:03:24.4624653Z drm_panel_orientation_quirks 32768 1 drm 2025-12-04T09:03:24.4625037Z backlight 24576 1 drm 2025-12-04T09:03:24.4625375Z i2c_core 110592 2 nvidia,drm 2025-12-04T09:03:24.4625736Z xt_conntrack 16384 1 2025-12-04T09:03:24.4626053Z nft_chain_nat 16384 3 2025-12-04T09:03:24.4626356Z xt_MASQUERADE 20480 1 2025-12-04T09:03:24.4626724Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-12-04T09:03:24.4627150Z nf_conntrack_netlink 57344 0 2025-12-04T09:03:24.4627742Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-12-04T09:03:24.4628486Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-12-04T09:03:24.4628891Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-12-04T09:03:24.4629259Z xfrm_user 57344 1 2025-12-04T09:03:24.4629575Z xfrm_algo 16384 1 xfrm_user 2025-12-04T09:03:24.4629933Z xt_addrtype 16384 2 2025-12-04T09:03:24.4630251Z nft_compat 20480 4 2025-12-04T09:03:24.4630627Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-12-04T09:03:24.4631135Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-12-04T09:03:24.4631606Z br_netfilter 36864 0 2025-12-04T09:03:24.4631945Z bridge 323584 1 br_netfilter 2025-12-04T09:03:24.4632297Z stp 16384 1 bridge 2025-12-04T09:03:24.4632645Z llc 16384 2 bridge,stp 2025-12-04T09:03:24.4633001Z overlay 167936 0 2025-12-04T09:03:24.4633294Z tls 139264 0 2025-12-04T09:03:24.4633597Z nls_ascii 16384 1 2025-12-04T09:03:24.4633919Z nls_cp437 20480 1 2025-12-04T09:03:24.4634227Z vfat 24576 1 2025-12-04T09:03:24.4634523Z fat 86016 1 vfat 2025-12-04T09:03:24.4634856Z sunrpc 700416 1 2025-12-04T09:03:24.4635171Z skx_edac_common 28672 0 2025-12-04T09:03:24.4635464Z i8042 45056 0 2025-12-04T09:03:24.4635779Z ghash_clmulni_intel 16384 0 2025-12-04T09:03:24.4636106Z serio 28672 3 i8042 2025-12-04T09:03:24.4636427Z ena 184320 0 2025-12-04T09:03:24.4636734Z button 24576 0 2025-12-04T09:03:24.4637075Z sch_fq_codel 20480 33 2025-12-04T09:03:24.4637525Z fuse 184320 1 2025-12-04T09:03:24.4638028Z dm_mod 188416 0 2025-12-04T09:03:24.4638497Z loop 36864 0 2025-12-04T09:03:24.4639098Z configfs 57344 1 2025-12-04T09:03:24.4639593Z dmi_sysfs 20480 0 2025-12-04T09:03:24.4640111Z crc32_pclmul 16384 0 2025-12-04T09:03:24.4640642Z crc32c_intel 24576 0 2025-12-04T09:03:24.4641158Z efivarfs 24576 1 2025-12-04T09:03:24.4641459Z + modinfo nvidia 2025-12-04T09:03:24.4642001Z filename: /lib/modules/6.1.150-174.273.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-12-04T09:03:24.4642913Z import_ns: DMA_BUF 2025-12-04T09:03:24.4643209Z alias: char-major-195-* 2025-12-04T09:03:24.4643526Z version: 580.82.07 2025-12-04T09:03:24.4643816Z supported: external 2025-12-04T09:03:24.4644097Z license: Dual MIT/GPL 2025-12-04T09:03:24.4644653Z firmware: nvidia/580.82.07/gsp_tu10x.bin 2025-12-04T09:03:24.4645372Z firmware: nvidia/580.82.07/gsp_ga10x.bin 2025-12-04T09:03:24.4645931Z srcversion: BA7240A71DCF7DC6FE88C1D 2025-12-04T09:03:24.4646489Z alias: of:N*T*Cnvidia,tegra264-displayC* 2025-12-04T09:03:24.4646912Z alias: of:N*T*Cnvidia,tegra264-display 2025-12-04T09:03:24.4647330Z alias: of:N*T*Cnvidia,tegra234-displayC* 2025-12-04T09:03:24.4647731Z alias: of:N*T*Cnvidia,tegra234-display 2025-12-04T09:03:24.4648190Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-12-04T09:03:24.4648853Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-12-04T09:03:24.4649287Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-12-04T09:03:24.4649662Z depends: i2c-core,drm 2025-12-04T09:03:24.4649968Z retpoline: Y 2025-12-04T09:03:24.4650215Z name: nvidia 2025-12-04T09:03:24.4650649Z vermagic: 6.1.150-174.273.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-12-04T09:03:24.4651219Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-12-04T09:03:24.4651753Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-12-04T09:03:24.4652247Z parm: NVreg_ResmanDebugLevel:int 2025-12-04T09:03:24.4652624Z parm: NVreg_RmLogonRC:int 2025-12-04T09:03:24.4652980Z parm: NVreg_ModifyDeviceFiles:int 2025-12-04T09:03:24.4653344Z parm: NVreg_DeviceFileUID:int 2025-12-04T09:03:24.4653708Z parm: NVreg_DeviceFileGID:int 2025-12-04T09:03:24.4654074Z parm: NVreg_DeviceFileMode:int 2025-12-04T09:03:24.4654494Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-12-04T09:03:24.4654959Z parm: NVreg_UsePageAttributeTable:int 2025-12-04T09:03:24.4655360Z parm: NVreg_EnablePCIeGen3:int 2025-12-04T09:03:24.4655708Z parm: NVreg_EnableMSI:int 2025-12-04T09:03:24.4656074Z parm: NVreg_EnableStreamMemOPs:int 2025-12-04T09:03:24.4656503Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-12-04T09:03:24.4656973Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-12-04T09:03:24.4657411Z parm: NVreg_EnableS0ixPowerManagement:int 2025-12-04T09:03:24.4657903Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-12-04T09:03:24.4658388Z parm: NVreg_DynamicPowerManagement:int 2025-12-04T09:03:24.4658870Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-12-04T09:03:24.4659363Z parm: NVreg_EnableGpuFirmware:int 2025-12-04T09:03:24.4659764Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-12-04T09:03:24.4660207Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-12-04T09:03:24.4660745Z parm: NVreg_EnableUserNUMAManagement:int 2025-12-04T09:03:24.4661140Z parm: NVreg_MemoryPoolSize:int 2025-12-04T09:03:24.4661517Z parm: NVreg_KMallocHeapMaxSize:int 2025-12-04T09:03:24.4661888Z parm: NVreg_VMallocHeapMaxSize:int 2025-12-04T09:03:24.4662265Z parm: NVreg_IgnoreMMIOCheck:int 2025-12-04T09:03:24.4662626Z parm: NVreg_NvLinkDisable:int 2025-12-04T09:03:24.4663014Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-12-04T09:03:24.4663430Z parm: NVreg_RegisterPCIDriver:int 2025-12-04T09:03:24.4663835Z parm: NVreg_RegisterPlatformDeviceDriver:int 2025-12-04T09:03:24.4664238Z parm: NVreg_EnableResizableBar:int 2025-12-04T09:03:24.4664624Z parm: NVreg_EnableDbgBreakpoint:int 2025-12-04T09:03:24.4665150Z parm: NVreg_EnableNonblockingOpen:int 2025-12-04T09:03:24.4665559Z parm: NVreg_CoherentGPUMemoryMode:charp 2025-12-04T09:03:24.4665944Z parm: NVreg_RegistryDwords:charp 2025-12-04T09:03:24.4666339Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-12-04T09:03:24.4666729Z parm: NVreg_RmMsg:charp 2025-12-04T09:03:24.4667050Z parm: NVreg_GpuBlacklist:charp 2025-12-04T09:03:24.4667537Z parm: NVreg_TemporaryFilePath:charp 2025-12-04T09:03:24.4668108Z parm: NVreg_ExcludedGpus:charp 2025-12-04T09:03:24.4668528Z parm: NVreg_DmaRemapPeerMmio:int 2025-12-04T09:03:24.4668935Z parm: NVreg_RmNvlinkBandwidth:charp 2025-12-04T09:03:24.4669475Z parm: NVreg_RmNvlinkBandwidthLinkCount:int 2025-12-04T09:03:24.4669908Z parm: NVreg_ImexChannelCount:int 2025-12-04T09:03:24.4670294Z parm: NVreg_CreateImexChannel0:int 2025-12-04T09:03:24.4670720Z parm: NVreg_GrdmaPciTopoCheckOverride:int 2025-12-04T09:03:24.4671145Z parm: rm_firmware_active:charp 2025-12-04T09:03:24.4671488Z + HAS_NVIDIA_DRIVER=0 2025-12-04T09:03:24.4671795Z ++ command -v nvidia-smi 2025-12-04T09:03:24.4672111Z + '[' -x /usr/bin/nvidia-smi ']' 2025-12-04T09:03:24.4672413Z + set +e 2025-12-04T09:03:24.4672793Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2025-12-04T09:03:27.5623808Z + INSTALLED_DRIVER_VERSION=580.82.07 2025-12-04T09:03:27.5624287Z + NVIDIA_SMI_STATUS=0 2025-12-04T09:03:27.5625822Z + '[' 0 -ne 0 ']' 2025-12-04T09:03:27.5626082Z + '[' 580.82.07 '!=' 580.82.07 ']' 2025-12-04T09:03:27.5626413Z + HAS_NVIDIA_DRIVER=1 2025-12-04T09:03:27.5627000Z + echo 'NVIDIA driver (580.82.07) has already been installed. Skipping NVIDIA driver installation' 2025-12-04T09:03:27.5627701Z + set -e 2025-12-04T09:03:27.5627924Z + '[' 1 -eq 0 ']' 2025-12-04T09:03:27.5628409Z NVIDIA driver (580.82.07) has already been installed. Skipping NVIDIA driver installation 2025-12-04T09:03:27.5629014Z + post_install_nvidia_driver_common 2025-12-04T09:03:27.5629662Z + sudo modprobe nvidia 2025-12-04T09:03:27.6699535Z + echo 'After installing NVIDIA driver' 2025-12-04T09:03:27.6700270Z + lspci 2025-12-04T09:03:27.6700539Z After installing NVIDIA driver 2025-12-04T09:03:27.6824297Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-12-04T09:03:27.6825537Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-12-04T09:03:27.6826268Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-12-04T09:03:27.6826939Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-12-04T09:03:27.6827692Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-12-04T09:03:27.6828353Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-12-04T09:03:27.6828989Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-12-04T09:03:27.6829583Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-12-04T09:03:27.6830159Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-12-04T09:03:27.6830718Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2025-12-04T09:03:27.6831347Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-12-04T09:03:27.6831862Z + lsmod 2025-12-04T09:03:27.6853556Z Module Size Used by 2025-12-04T09:03:27.6854013Z nvidia_uvm 1925120 0 2025-12-04T09:03:27.6854400Z nvidia 14286848 1 nvidia_uvm 2025-12-04T09:03:27.6854747Z drm 602112 1 nvidia 2025-12-04T09:03:27.6855117Z drm_panel_orientation_quirks 32768 1 drm 2025-12-04T09:03:27.6855510Z backlight 24576 1 drm 2025-12-04T09:03:27.6855849Z i2c_core 110592 2 nvidia,drm 2025-12-04T09:03:27.6856184Z xt_conntrack 16384 1 2025-12-04T09:03:27.6856495Z nft_chain_nat 16384 3 2025-12-04T09:03:27.6857035Z xt_MASQUERADE 20480 1 2025-12-04T09:03:27.6857400Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-12-04T09:03:27.6857790Z nf_conntrack_netlink 57344 0 2025-12-04T09:03:27.6858271Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-12-04T09:03:27.6858808Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-12-04T09:03:27.6859177Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-12-04T09:03:27.6859531Z xfrm_user 57344 1 2025-12-04T09:03:27.6859848Z xfrm_algo 16384 1 xfrm_user 2025-12-04T09:03:27.6860180Z xt_addrtype 16384 2 2025-12-04T09:03:27.6860489Z nft_compat 20480 4 2025-12-04T09:03:27.6860978Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-12-04T09:03:27.6861486Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-12-04T09:03:27.6861931Z br_netfilter 36864 0 2025-12-04T09:03:27.6862259Z bridge 323584 1 br_netfilter 2025-12-04T09:03:27.6862621Z stp 16384 1 bridge 2025-12-04T09:03:27.6862948Z llc 16384 2 bridge,stp 2025-12-04T09:03:27.6863290Z overlay 167936 0 2025-12-04T09:03:27.6863586Z tls 139264 0 2025-12-04T09:03:27.6863872Z nls_ascii 16384 1 2025-12-04T09:03:27.6864169Z nls_cp437 20480 1 2025-12-04T09:03:27.6864461Z vfat 24576 1 2025-12-04T09:03:27.6864749Z fat 86016 1 vfat 2025-12-04T09:03:27.6865068Z sunrpc 700416 1 2025-12-04T09:03:27.6865369Z skx_edac_common 28672 0 2025-12-04T09:03:27.6865652Z i8042 45056 0 2025-12-04T09:03:27.6865950Z ghash_clmulni_intel 16384 0 2025-12-04T09:03:27.6866270Z serio 28672 3 i8042 2025-12-04T09:03:27.6866589Z ena 184320 0 2025-12-04T09:03:27.6866868Z button 24576 0 2025-12-04T09:03:27.6867170Z sch_fq_codel 20480 33 2025-12-04T09:03:27.6867613Z fuse 184320 1 2025-12-04T09:03:27.6867894Z dm_mod 188416 0 2025-12-04T09:03:27.6868357Z loop 36864 0 2025-12-04T09:03:27.6868667Z configfs 57344 1 2025-12-04T09:03:27.6868965Z dmi_sysfs 20480 0 2025-12-04T09:03:27.6869282Z crc32_pclmul 16384 0 2025-12-04T09:03:27.6869593Z crc32c_intel 24576 0 2025-12-04T09:03:27.6869888Z efivarfs 24576 1 2025-12-04T09:03:27.6870217Z + modinfo nvidia 2025-12-04T09:03:27.6870695Z filename: /lib/modules/6.1.150-174.273.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-12-04T09:03:27.6871254Z import_ns: DMA_BUF 2025-12-04T09:03:27.6871544Z alias: char-major-195-* 2025-12-04T09:03:27.6871878Z version: 580.82.07 2025-12-04T09:03:27.6872176Z supported: external 2025-12-04T09:03:27.6872481Z license: Dual MIT/GPL 2025-12-04T09:03:27.6872815Z firmware: nvidia/580.82.07/gsp_tu10x.bin 2025-12-04T09:03:27.6873232Z firmware: nvidia/580.82.07/gsp_ga10x.bin 2025-12-04T09:03:27.6873629Z srcversion: BA7240A71DCF7DC6FE88C1D 2025-12-04T09:03:27.6874024Z alias: of:N*T*Cnvidia,tegra264-displayC* 2025-12-04T09:03:27.6874457Z alias: of:N*T*Cnvidia,tegra264-display 2025-12-04T09:03:27.6874883Z alias: of:N*T*Cnvidia,tegra234-displayC* 2025-12-04T09:03:27.6875298Z alias: of:N*T*Cnvidia,tegra234-display 2025-12-04T09:03:27.6875712Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-12-04T09:03:27.6876119Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-12-04T09:03:27.6876545Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-12-04T09:03:27.6876931Z depends: i2c-core,drm 2025-12-04T09:03:27.6877237Z retpoline: Y 2025-12-04T09:03:27.6877505Z name: nvidia 2025-12-04T09:03:27.6877953Z vermagic: 6.1.150-174.273.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-12-04T09:03:27.6878545Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-12-04T09:03:27.6879185Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-12-04T09:03:27.6879711Z parm: NVreg_ResmanDebugLevel:int 2025-12-04T09:03:27.6880196Z parm: NVreg_RmLogonRC:int 2025-12-04T09:03:27.6880545Z parm: NVreg_ModifyDeviceFiles:int 2025-12-04T09:03:27.6880928Z parm: NVreg_DeviceFileUID:int 2025-12-04T09:03:27.6881292Z parm: NVreg_DeviceFileGID:int 2025-12-04T09:03:27.6881645Z parm: NVreg_DeviceFileMode:int 2025-12-04T09:03:27.6882080Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-12-04T09:03:27.6882543Z parm: NVreg_UsePageAttributeTable:int 2025-12-04T09:03:27.6882943Z parm: NVreg_EnablePCIeGen3:int 2025-12-04T09:03:27.6883370Z parm: NVreg_EnableMSI:int 2025-12-04T09:03:27.6883736Z parm: NVreg_EnableStreamMemOPs:int 2025-12-04T09:03:27.6884167Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-12-04T09:03:27.6884625Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-12-04T09:03:27.6885079Z parm: NVreg_EnableS0ixPowerManagement:int 2025-12-04T09:03:27.6885673Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-12-04T09:03:27.6886134Z parm: NVreg_DynamicPowerManagement:int 2025-12-04T09:03:27.6886615Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-12-04T09:03:27.6887086Z parm: NVreg_EnableGpuFirmware:int 2025-12-04T09:03:27.6887474Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-12-04T09:03:27.6887886Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-12-04T09:03:27.6888316Z parm: NVreg_EnableUserNUMAManagement:int 2025-12-04T09:03:27.6888712Z parm: NVreg_MemoryPoolSize:int 2025-12-04T09:03:27.6889073Z parm: NVreg_KMallocHeapMaxSize:int 2025-12-04T09:03:27.6889458Z parm: NVreg_VMallocHeapMaxSize:int 2025-12-04T09:03:27.6889835Z parm: NVreg_IgnoreMMIOCheck:int 2025-12-04T09:03:27.6890191Z parm: NVreg_NvLinkDisable:int 2025-12-04T09:03:27.6890592Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-12-04T09:03:27.6891007Z parm: NVreg_RegisterPCIDriver:int 2025-12-04T09:03:27.6891406Z parm: NVreg_RegisterPlatformDeviceDriver:int 2025-12-04T09:03:27.6891823Z parm: NVreg_EnableResizableBar:int 2025-12-04T09:03:27.6892209Z parm: NVreg_EnableDbgBreakpoint:int 2025-12-04T09:03:27.6892611Z parm: NVreg_EnableNonblockingOpen:int 2025-12-04T09:03:27.6893005Z parm: NVreg_CoherentGPUMemoryMode:charp 2025-12-04T09:03:27.6893399Z parm: NVreg_RegistryDwords:charp 2025-12-04T09:03:27.6893788Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-12-04T09:03:27.6894161Z parm: NVreg_RmMsg:charp 2025-12-04T09:03:27.6894491Z parm: NVreg_GpuBlacklist:charp 2025-12-04T09:03:27.6894867Z parm: NVreg_TemporaryFilePath:charp 2025-12-04T09:03:27.6895231Z parm: NVreg_ExcludedGpus:charp 2025-12-04T09:03:27.6895600Z parm: NVreg_DmaRemapPeerMmio:int 2025-12-04T09:03:27.6895980Z parm: NVreg_RmNvlinkBandwidth:charp 2025-12-04T09:03:27.6896392Z parm: NVreg_RmNvlinkBandwidthLinkCount:int 2025-12-04T09:03:27.6896784Z parm: NVreg_ImexChannelCount:int 2025-12-04T09:03:27.6897330Z parm: NVreg_CreateImexChannel0:int 2025-12-04T09:03:27.6897743Z parm: NVreg_GrdmaPciTopoCheckOverride:int 2025-12-04T09:03:27.6898138Z parm: rm_firmware_active:charp 2025-12-04T09:03:27.6898479Z + set +e 2025-12-04T09:03:27.6898702Z + nvidia-smi 2025-12-04T09:03:29.4875139Z Thu Dec 4 09:03:29 2025 2025-12-04T09:03:29.4875677Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:03:29.4876323Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T09:03:29.4876937Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:03:29.4877823Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T09:03:29.4878496Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T09:03:29.4879038Z | | | MIG M. | 2025-12-04T09:03:29.4879446Z |=========================================+========================+======================| 2025-12-04T09:03:29.5259088Z | 0 Tesla T4 Off | 00000000:00:1B.0 Off | 0 | 2025-12-04T09:03:29.5259673Z | N/A 28C P0 26W / 70W | 0MiB / 15360MiB | 2% Default | 2025-12-04T09:03:29.5260373Z | | | N/A | 2025-12-04T09:03:29.5260855Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:03:29.5261406Z | 1 Tesla T4 Off | 00000000:00:1C.0 Off | 0 | 2025-12-04T09:03:29.5262000Z | N/A 29C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:03:29.5262495Z | | | N/A | 2025-12-04T09:03:29.5262978Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:03:29.5263510Z | 2 Tesla T4 Off | 00000000:00:1D.0 Off | 0 | 2025-12-04T09:03:29.5264020Z | N/A 29C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:03:29.5264470Z | | | N/A | 2025-12-04T09:03:29.5264948Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:03:29.5265478Z | 3 Tesla T4 Off | 00000000:00:1E.0 Off | 0 | 2025-12-04T09:03:29.5265991Z | N/A 30C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:03:29.5266434Z | | | N/A | 2025-12-04T09:03:29.5266910Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:03:29.5267387Z 2025-12-04T09:03:29.5267595Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:03:29.5268302Z | Processes: | 2025-12-04T09:03:29.5268849Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T09:03:29.5269370Z | ID ID Usage | 2025-12-04T09:03:29.5269800Z |=========================================================================================| 2025-12-04T09:03:29.5284052Z | No running processes found | 2025-12-04T09:03:29.5284664Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:03:31.1832374Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-12-04T09:03:32.9670393Z Tesla T4 2025-12-04T09:03:34.2825988Z + NVIDIA_SMI_STATUS=0 2025-12-04T09:03:34.2826362Z + '[' 0 -eq 0 ']' 2025-12-04T09:03:34.2826653Z + echo 'INFO: Ignoring allowed status 0' 2025-12-04T09:03:34.2827016Z + set -e 2025-12-04T09:03:34.2827365Z INFO: Ignoring allowed status 0 2025-12-04T09:03:34.2832714Z == Installing nvidia container toolkit for amzn2023 == 2025-12-04T09:03:34.2836580Z + sudo yum install -y yum-utils 2025-12-04T09:03:34.7467177Z Last metadata expiration check: 0:07:31 ago on Thu Dec 4 08:56:03 2025. 2025-12-04T09:03:34.7779029Z Package dnf-utils-4.3.0-13.amzn2023.0.5.noarch is already installed. 2025-12-04T09:03:34.8383017Z Dependencies resolved. 2025-12-04T09:03:34.8685443Z Nothing to do. 2025-12-04T09:03:34.8685846Z Complete! 2025-12-04T09:03:35.0927545Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2025-12-04T09:03:35.0928314Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-12-04T09:03:35.0929415Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-12-04T09:03:35.4179244Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-12-04T09:03:35.4679334Z + sudo yum install -y nvidia-container-toolkit-1.17.8 libnvidia-container-tools-1.17.8 libnvidia-container1-1.17.8 nvidia-container-toolkit-base-1.17.8 2025-12-04T09:03:36.0435322Z nvidia-container-toolkit 21 kB/s | 833 B 00:00 2025-12-04T09:03:36.1387585Z Dependencies resolved. 2025-12-04T09:03:36.1693984Z ================================================================================ 2025-12-04T09:03:36.1694658Z Package Arch Version Repository Size 2025-12-04T09:03:36.1695131Z ================================================================================ 2025-12-04T09:03:36.1695502Z Downgrading: 2025-12-04T09:03:36.1695936Z libnvidia-container-tools x86_64 1.17.8-1 nvidia-container-toolkit 40 k 2025-12-04T09:03:36.1696624Z libnvidia-container1 x86_64 1.17.8-1 nvidia-container-toolkit 1.0 M 2025-12-04T09:03:36.1697301Z nvidia-container-toolkit x86_64 1.17.8-1 nvidia-container-toolkit 1.2 M 2025-12-04T09:03:36.1698000Z nvidia-container-toolkit-base x86_64 1.17.8-1 nvidia-container-toolkit 5.8 M 2025-12-04T09:03:36.1698444Z 2025-12-04T09:03:36.1698565Z Transaction Summary 2025-12-04T09:03:36.1698863Z ================================================================================ 2025-12-04T09:03:36.1699238Z Downgrade 4 Packages 2025-12-04T09:03:36.1699414Z 2025-12-04T09:03:36.1699535Z Total download size: 8.0 M 2025-12-04T09:03:36.1699848Z Downloading Packages: 2025-12-04T09:03:36.3477883Z (1/4): libnvidia-container-tools-1.17.8-1.x86_6 229 kB/s | 40 kB 00:00 2025-12-04T09:03:36.4118453Z (2/4): libnvidia-container1-1.17.8-1.x86_64.rpm 4.1 MB/s | 1.0 MB 00:00 2025-12-04T09:03:36.4651857Z (3/4): nvidia-container-toolkit-1.17.8-1.x86_64 4.2 MB/s | 1.2 MB 00:00 2025-12-04T09:03:36.6821326Z (4/4): nvidia-container-toolkit-base-1.17.8-1.x 17 MB/s | 5.8 MB 00:00 2025-12-04T09:03:36.6828921Z -------------------------------------------------------------------------------- 2025-12-04T09:03:36.6832442Z Total 16 MB/s | 8.0 MB 00:00 2025-12-04T09:03:36.6835105Z Running transaction check 2025-12-04T09:03:36.6984013Z Transaction check succeeded. 2025-12-04T09:03:36.6984532Z Running transaction test 2025-12-04T09:03:36.7501767Z Transaction test succeeded. 2025-12-04T09:03:36.7502353Z Running transaction 2025-12-04T09:03:37.6458039Z Preparing : 1/1 2025-12-04T09:03:37.9072168Z Downgrading : nvidia-container-toolkit-base-1.17.8-1.x86_64 1/8 2025-12-04T09:03:37.9188145Z Downgrading : libnvidia-container1-1.17.8-1.x86_64 2/8 2025-12-04T09:03:37.9473941Z Running scriptlet: libnvidia-container1-1.17.8-1.x86_64 2/8 2025-12-04T09:03:38.0773865Z Downgrading : libnvidia-container-tools-1.17.8-1.x86_64 3/8 2025-12-04T09:03:38.0923337Z Downgrading : nvidia-container-toolkit-1.17.8-1.x86_64 4/8 2025-12-04T09:03:38.1367163Z Running scriptlet: nvidia-container-toolkit-1.17.8-1.x86_64 4/8 2025-12-04T09:03:38.1419221Z Running scriptlet: nvidia-container-toolkit-1.18.1-1.x86_64 5/8 2025-12-04T09:03:38.1420150Z Cleanup : nvidia-container-toolkit-1.18.1-1.x86_64 5/8 2025-12-04T09:03:38.1858764Z Running scriptlet: nvidia-container-toolkit-1.18.1-1.x86_64 5/8 2025-12-04T09:03:38.1902207Z Running scriptlet: libnvidia-container-tools-1.18.1-1.x86_64 6/8 2025-12-04T09:03:38.1903184Z Cleanup : libnvidia-container-tools-1.18.1-1.x86_64 6/8 2025-12-04T09:03:38.2269662Z Running scriptlet: libnvidia-container-tools-1.18.1-1.x86_64 6/8 2025-12-04T09:03:38.2318116Z Running scriptlet: libnvidia-container1-1.18.1-1.x86_64 7/8 2025-12-04T09:03:38.2319075Z Cleanup : libnvidia-container1-1.18.1-1.x86_64 7/8 2025-12-04T09:03:38.3014236Z Running scriptlet: libnvidia-container1-1.18.1-1.x86_64 7/8 2025-12-04T09:03:38.3062808Z Running scriptlet: nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8 2025-12-04T09:03:38.3063760Z Cleanup : nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8 2025-12-04T09:03:38.3469185Z Running scriptlet: nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8 2025-12-04T09:03:38.3997452Z Running scriptlet: nvidia-container-toolkit-1.17.8-1.x86_64 8/8 2025-12-04T09:04:49.0452863Z Running scriptlet: nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8 2025-12-04T09:04:49.0453870Z Verifying : libnvidia-container-tools-1.17.8-1.x86_64 1/8 2025-12-04T09:04:49.0454595Z Verifying : libnvidia-container-tools-1.18.1-1.x86_64 2/8 2025-12-04T09:04:49.0455493Z Verifying : libnvidia-container1-1.17.8-1.x86_64 3/8 2025-12-04T09:04:49.0456229Z Verifying : libnvidia-container1-1.18.1-1.x86_64 4/8 2025-12-04T09:04:49.0456993Z Verifying : nvidia-container-toolkit-1.17.8-1.x86_64 5/8 2025-12-04T09:04:49.0457738Z Verifying : nvidia-container-toolkit-1.18.1-1.x86_64 6/8 2025-12-04T09:04:49.0458489Z Verifying : nvidia-container-toolkit-base-1.17.8-1.x86_64 7/8 2025-12-04T09:04:49.1975775Z Verifying : nvidia-container-toolkit-base-1.18.1-1.x86_64 8/8================================================================================ 2025-12-04T09:04:49.1976684Z WARNING: 2025-12-04T09:04:49.1977002Z A newer release of "Amazon Linux" is available. 2025-12-04T09:04:49.1977413Z 2025-12-04T09:04:49.1977547Z Available Versions: 2025-12-04T09:04:49.1977750Z 2025-12-04T09:04:49.1977922Z Version 2023.9.20250929: 2025-12-04T09:04:49.1978396Z Run the following command to upgrade to 2023.9.20250929: 2025-12-04T09:04:49.1978744Z 2025-12-04T09:04:49.1978913Z dnf upgrade --releasever=2023.9.20250929 2025-12-04T09:04:49.1979235Z 2025-12-04T09:04:49.1979361Z Release notes: 2025-12-04T09:04:49.1979987Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20250929.html 2025-12-04T09:04:49.1980484Z 2025-12-04T09:04:49.1980651Z Version 2023.9.20251014: 2025-12-04T09:04:49.1981060Z Run the following command to upgrade to 2023.9.20251014: 2025-12-04T09:04:49.1981437Z 2025-12-04T09:04:49.1981627Z dnf upgrade --releasever=2023.9.20251014 2025-12-04T09:04:49.1981916Z 2025-12-04T09:04:49.1982109Z Release notes: 2025-12-04T09:04:49.1982638Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251014.html 2025-12-04T09:04:49.1983137Z 2025-12-04T09:04:49.1983295Z Version 2023.9.20251020: 2025-12-04T09:04:49.1983756Z Run the following command to upgrade to 2023.9.20251020: 2025-12-04T09:04:49.1984086Z 2025-12-04T09:04:49.1984288Z dnf upgrade --releasever=2023.9.20251020 2025-12-04T09:04:49.1984547Z 2025-12-04T09:04:49.1984701Z Release notes: 2025-12-04T09:04:49.1985264Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251020.html 2025-12-04T09:04:49.1985753Z 2025-12-04T09:04:49.1985924Z Version 2023.9.20251027: 2025-12-04T09:04:49.1986396Z Run the following command to upgrade to 2023.9.20251027: 2025-12-04T09:04:49.1986727Z 2025-12-04T09:04:49.1987160Z dnf upgrade --releasever=2023.9.20251027 2025-12-04T09:04:49.1987597Z 2025-12-04T09:04:49.1987889Z Release notes: 2025-12-04T09:04:49.1988502Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251027.html 2025-12-04T09:04:49.1989029Z 2025-12-04T09:04:49.1989205Z Version 2023.9.20251105: 2025-12-04T09:04:49.1989631Z Run the following command to upgrade to 2023.9.20251105: 2025-12-04T09:04:49.1990041Z 2025-12-04T09:04:49.1990215Z dnf upgrade --releasever=2023.9.20251105 2025-12-04T09:04:49.1990510Z 2025-12-04T09:04:49.1990678Z Release notes: 2025-12-04T09:04:49.1991205Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251105.html 2025-12-04T09:04:49.1991754Z 2025-12-04T09:04:49.1992058Z Version 2023.9.20251110: 2025-12-04T09:04:49.1992522Z Run the following command to upgrade to 2023.9.20251110: 2025-12-04T09:04:49.1992865Z 2025-12-04T09:04:49.1993049Z dnf upgrade --releasever=2023.9.20251110 2025-12-04T09:04:49.1993394Z 2025-12-04T09:04:49.1993536Z Release notes: 2025-12-04T09:04:49.1994106Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251110.html 2025-12-04T09:04:49.1994638Z 2025-12-04T09:04:49.1994751Z Version 2023.9.20251117: 2025-12-04T09:04:49.1995283Z Run the following command to upgrade to 2023.9.20251117: 2025-12-04T09:04:49.1995620Z 2025-12-04T09:04:49.1995780Z dnf upgrade --releasever=2023.9.20251117 2025-12-04T09:04:49.1996104Z 2025-12-04T09:04:49.1996224Z Release notes: 2025-12-04T09:04:49.1996842Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251117.html 2025-12-04T09:04:49.1997330Z 2025-12-04T09:04:49.1997530Z ================================================================================ 2025-12-04T09:04:49.2588589Z 2025-12-04T09:04:49.2588970Z 2025-12-04T09:04:49.2589107Z Downgraded: 2025-12-04T09:04:49.2589858Z libnvidia-container-tools-1.17.8-1.x86_64 2025-12-04T09:04:49.2590749Z libnvidia-container1-1.17.8-1.x86_64 2025-12-04T09:04:49.2591486Z nvidia-container-toolkit-1.17.8-1.x86_64 2025-12-04T09:04:49.2592278Z nvidia-container-toolkit-base-1.17.8-1.x86_64 2025-12-04T09:04:49.2592774Z 2025-12-04T09:04:49.2593021Z Complete! 2025-12-04T09:04:49.3341633Z + sudo systemctl restart docker 2025-12-04T09:04:58.1509192Z Thu Dec 4 09:04:58 2025 2025-12-04T09:04:58.1509980Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:04:58.1510690Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T09:04:58.1511475Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:04:58.1512257Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T09:04:58.1513000Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T09:04:58.1513666Z | | | MIG M. | 2025-12-04T09:04:58.1514223Z |=========================================+========================+======================| 2025-12-04T09:04:58.1914965Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2025-12-04T09:04:58.1915723Z | N/A 28C P0 26W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:04:58.1916296Z | | | N/A | 2025-12-04T09:04:58.1916939Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:04:58.1917613Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2025-12-04T09:04:58.1918215Z | N/A 29C P0 24W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:04:58.1919040Z | | | N/A | 2025-12-04T09:04:58.1919772Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:04:58.1920435Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2025-12-04T09:04:58.1920998Z | N/A 28C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:04:58.1921569Z | | | N/A | 2025-12-04T09:04:58.1922249Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:04:58.1923038Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2025-12-04T09:04:58.1923828Z | N/A 30C P0 25W / 70W | 0MiB / 15360MiB | 9% Default | 2025-12-04T09:04:58.1924637Z | | | N/A | 2025-12-04T09:04:58.1925242Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:04:58.1936443Z 2025-12-04T09:04:58.1936708Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:04:58.1937264Z | Processes: | 2025-12-04T09:04:58.1937803Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T09:04:58.1938284Z | ID ID Usage | 2025-12-04T09:04:58.1938883Z |=========================================================================================| 2025-12-04T09:04:58.1940122Z | No running processes found | 2025-12-04T09:04:58.1940709Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:04:58.5906447Z Unable to find image 'public.ecr.aws/docker/library/python:3.13' locally 2025-12-04T09:04:58.7918460Z 3.13: Pulling from docker/library/python 2025-12-04T09:04:58.8870657Z 53c88f1dfeb7: Pulling fs layer 2025-12-04T09:04:58.8871029Z eae668646f44: Pulling fs layer 2025-12-04T09:04:58.8871362Z ff2e6e687b6c: Pulling fs layer 2025-12-04T09:04:58.8871697Z 7c40a3faff76: Pulling fs layer 2025-12-04T09:04:58.8872030Z 967a3b1c8fef: Pulling fs layer 2025-12-04T09:04:58.8872356Z a64e1a44f22a: Pulling fs layer 2025-12-04T09:04:58.8872671Z 52655f8a5bcc: Pulling fs layer 2025-12-04T09:04:58.8872989Z 52655f8a5bcc: Waiting 2025-12-04T09:04:58.8873275Z a64e1a44f22a: Waiting 2025-12-04T09:04:58.8873679Z 967a3b1c8fef: Waiting 2025-12-04T09:04:58.8873957Z 7c40a3faff76: Waiting 2025-12-04T09:04:59.0332560Z eae668646f44: Verifying Checksum 2025-12-04T09:04:59.0333012Z eae668646f44: Download complete 2025-12-04T09:04:59.1226233Z 53c88f1dfeb7: Verifying Checksum 2025-12-04T09:04:59.1226893Z 53c88f1dfeb7: Download complete 2025-12-04T09:04:59.2019482Z 967a3b1c8fef: Verifying Checksum 2025-12-04T09:04:59.2020158Z 967a3b1c8fef: Download complete 2025-12-04T09:04:59.2323366Z ff2e6e687b6c: Verifying Checksum 2025-12-04T09:04:59.2324362Z ff2e6e687b6c: Download complete 2025-12-04T09:04:59.2596800Z 52655f8a5bcc: Download complete 2025-12-04T09:04:59.3479628Z a64e1a44f22a: Verifying Checksum 2025-12-04T09:04:59.3480334Z a64e1a44f22a: Download complete 2025-12-04T09:05:00.0678398Z 7c40a3faff76: Verifying Checksum 2025-12-04T09:05:00.0678825Z 7c40a3faff76: Download complete 2025-12-04T09:05:00.3323902Z 53c88f1dfeb7: Pull complete 2025-12-04T09:05:00.8499300Z eae668646f44: Pull complete 2025-12-04T09:05:02.5550705Z ff2e6e687b6c: Pull complete 2025-12-04T09:05:07.5842274Z 7c40a3faff76: Pull complete 2025-12-04T09:05:07.7703399Z 967a3b1c8fef: Pull complete 2025-12-04T09:05:08.3259623Z a64e1a44f22a: Pull complete 2025-12-04T09:05:08.3407337Z 52655f8a5bcc: Pull complete 2025-12-04T09:05:08.3488687Z Digest: sha256:3f986299a7b8b44b0d8cf9bda2b22361ce5c3058ef5d7cb17fb7452506680ab0 2025-12-04T09:05:08.3513289Z Status: Downloaded newer image for public.ecr.aws/docker/library/python:3.13 2025-12-04T09:05:17.1594181Z Thu Dec 4 09:05:17 2025 2025-12-04T09:05:17.1594861Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:05:17.1595747Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T09:05:17.1596605Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:05:17.1597492Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T09:05:17.1598728Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T09:05:17.1599605Z | | | MIG M. | 2025-12-04T09:05:17.1600283Z |=========================================+========================+======================| 2025-12-04T09:05:17.2199172Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2025-12-04T09:05:17.2200086Z | N/A 27C P8 15W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:05:17.2200746Z | | | N/A | 2025-12-04T09:05:17.2201463Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:05:17.2202205Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2025-12-04T09:05:17.2202939Z | N/A 28C P8 13W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:05:17.2203629Z | | | N/A | 2025-12-04T09:05:17.2204281Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:05:17.2205024Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2025-12-04T09:05:17.2205770Z | N/A 27C P8 13W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:05:17.2206489Z | | | N/A | 2025-12-04T09:05:17.2207149Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:05:17.2207875Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2025-12-04T09:05:17.2208668Z | N/A 29C P8 13W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T09:05:17.2209439Z | | | N/A | 2025-12-04T09:05:17.2210279Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T09:05:17.2210837Z 2025-12-04T09:05:17.2211125Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:05:17.2211854Z | Processes: | 2025-12-04T09:05:17.2212596Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T09:05:17.2213510Z | ID ID Usage | 2025-12-04T09:05:17.2214252Z |=========================================================================================| 2025-12-04T09:05:17.2230436Z | No running processes found | 2025-12-04T09:05:17.2231366Z +-----------------------------------------------------------------------------------------+ 2025-12-04T09:05:18.6222696Z Command completed after 1 attempt(s). 2025-12-04T09:05:18.6319144Z Prepare all required actions 2025-12-04T09:05:18.6353246Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-12-04T09:05:18.6353636Z with: 2025-12-04T09:05:18.6354291Z github-token: *** 2025-12-04T09:05:18.6354562Z env: 2025-12-04T09:05:18.6354798Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:18.6355112Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:18.6355482Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:18.6355904Z ##[endgroup] 2025-12-04T09:05:18.6371065Z ##[group]Run set -eux 2025-12-04T09:05:18.6371341Z set -eux 2025-12-04T09:05:18.6371827Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-12-04T09:05:18.6381984Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:18.6382363Z env: 2025-12-04T09:05:18.6382591Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:18.6382872Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:18.6383241Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:18.6383776Z GITHUB_TOKEN: *** 2025-12-04T09:05:18.6384019Z ##[endgroup] 2025-12-04T09:05:18.6414897Z + python3 .github/scripts/get_workflow_job_id.py 19922768520 i-0452b6a3e0f4cdcdb 2025-12-04T09:05:20.0343413Z Setting output job-id=57116084912 2025-12-04T09:05:20.0344349Z Setting output job-name=linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T09:05:20.0454472Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T09:05:20.0455249Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-12-04T09:05:20.0456258Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-12-04T09:05:20.0457152Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-12-04T09:05:20.0463753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:20.0464151Z env: 2025-12-04T09:05:20.0464400Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:20.0464665Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:20.0464995Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:20.0465369Z JOB_ID: 57116084912 2025-12-04T09:05:20.0465994Z JOB_NAME: linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T09:05:20.0466639Z WORKFLOW_NAME: trunk 2025-12-04T09:05:20.0466891Z WORKFLOW_RUN_ID: 19922768520 2025-12-04T09:05:20.0467180Z MONITOR_LOG_INTERVAL: 5 2025-12-04T09:05:20.0467582Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-12-04T09:05:20.0468105Z ##[endgroup] 2025-12-04T09:05:20.3553647Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:05:20.7370328Z Collecting psutil==5.9.8 2025-12-04T09:05:20.7543809Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-12-04T09:05:20.8287358Z Collecting dataclasses_json==0.6.7 2025-12-04T09:05:20.8323336Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-12-04T09:05:20.8607114Z Collecting nvidia-ml-py==11.525.84 2025-12-04T09:05:20.8643708Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-12-04T09:05:20.8972862Z Collecting typing-inspect<1,>=0.4.0 2025-12-04T09:05:20.9006321Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-12-04T09:05:21.0348829Z Collecting marshmallow<4.0.0,>=3.18.0 2025-12-04T09:05:21.0384322Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-12-04T09:05:21.0950055Z Collecting packaging>=17.0 2025-12-04T09:05:21.0991272Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-12-04T09:05:21.1535083Z Collecting typing-extensions>=3.7.4 2025-12-04T09:05:21.1575023Z Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB) 2025-12-04T09:05:21.1762906Z Collecting mypy-extensions>=0.3.0 2025-12-04T09:05:21.1796208Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-12-04T09:05:21.2834585Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-12-04T09:05:21.5644517Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0 2025-12-04T09:05:21.7512214Z Prepare all required actions 2025-12-04T09:05:21.7512645Z Getting action download info 2025-12-04T09:05:21.9434226Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-12-04T09:05:22.1688499Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-12-04T09:05:22.4724397Z ##[group]Run ./.github/actions/download-build-artifacts 2025-12-04T09:05:22.4724832Z with: 2025-12-04T09:05:22.4725106Z name: linux-jammy-cuda12.8-py3.10-gcc11 2025-12-04T09:05:22.4725492Z s3-bucket: gha-artifacts 2025-12-04T09:05:22.4725819Z env: 2025-12-04T09:05:22.4726051Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:22.4726359Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:22.4726730Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:22.4727133Z ##[endgroup] 2025-12-04T09:05:22.4759103Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T09:05:22.4759612Z with: 2025-12-04T09:05:22.4759974Z name: linux-jammy-cuda12.8-py3.10-gcc11 2025-12-04T09:05:22.4760368Z s3-bucket: gha-artifacts 2025-12-04T09:05:22.4760689Z region: us-east-1 2025-12-04T09:05:22.4760946Z env: 2025-12-04T09:05:22.4761168Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:22.4761467Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:22.4761821Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:22.4762230Z ##[endgroup] 2025-12-04T09:05:22.9994766Z (node:62802) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T09:05:22.9995357Z 2025-12-04T09:05:22.9995626Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T09:05:22.9996246Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T09:05:22.9996908Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T09:05:23.2767429Z Found 1 objects with prefix pytorch/pytorch/19922768520/linux-jammy-cuda12.8-py3.10-gcc11/ 2025-12-04T09:05:23.2768300Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T09:05:31.5309327Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-12-04T09:05:31.5315258Z Artifact download has finished successfully 2025-12-04T09:05:31.5570204Z ##[group]Run unzip -o artifacts.zip 2025-12-04T09:05:31.5570557Z unzip -o artifacts.zip 2025-12-04T09:05:31.5576464Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:31.5576864Z env: 2025-12-04T09:05:31.5577080Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:31.5577390Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:31.5577716Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:31.5578073Z ##[endgroup] 2025-12-04T09:05:31.5655462Z Archive: artifacts.zip 2025-12-04T09:05:31.5655867Z creating: dist/ 2025-12-04T09:05:31.5790874Z inflating: dist/.ninja_log 2025-12-04T09:05:34.1011661Z inflating: dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T09:05:34.1012404Z creating: build/ 2025-12-04T09:05:34.1012711Z creating: build/custom_test_artifacts/ 2025-12-04T09:05:34.1013167Z creating: build/custom_test_artifacts/custom-op-build/ 2025-12-04T09:05:34.1013717Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-12-04T09:05:34.1014394Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:05:34.1019285Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:05:34.1020058Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/ 2025-12-04T09:05:34.1021084Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:05:34.1021907Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:05:34.1022878Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:05:34.1024188Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:05:34.1025143Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:05:34.1026048Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:05:34.1026917Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:05:34.1027876Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:05:34.1028860Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:05:34.1030132Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:05:34.1031077Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:05:34.1032617Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:05:34.1034508Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:05:34.1035453Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-12-04T09:05:34.1036311Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-12-04T09:05:34.1092019Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-12-04T09:05:34.1150115Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-12-04T09:05:34.1151393Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-12-04T09:05:34.1210083Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-12-04T09:05:34.1211467Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-12-04T09:05:34.1212685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-12-04T09:05:34.1213925Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-12-04T09:05:34.1215127Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-12-04T09:05:34.1216300Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-12-04T09:05:34.1217486Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-12-04T09:05:34.1218649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-12-04T09:05:34.1219777Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-12-04T09:05:34.1220864Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-12-04T09:05:34.1221918Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-12-04T09:05:34.1223158Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-12-04T09:05:34.1224585Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-12-04T09:05:34.1225763Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-12-04T09:05:34.1226813Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-12-04T09:05:34.1298870Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-12-04T09:05:34.1299819Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-12-04T09:05:34.1376415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-12-04T09:05:34.1377503Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:05:34.1378230Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2025-12-04T09:05:34.1378969Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-12-04T09:05:34.1379739Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-12-04T09:05:34.1380579Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-12-04T09:05:34.1381545Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-12-04T09:05:34.1382474Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-12-04T09:05:34.1383339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-12-04T09:05:34.1384218Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-12-04T09:05:34.1385120Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-12-04T09:05:34.1386019Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-12-04T09:05:34.1386916Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-12-04T09:05:34.1388088Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-12-04T09:05:34.1403957Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-12-04T09:05:34.1594015Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-12-04T09:05:34.1594906Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-12-04T09:05:34.1595831Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-12-04T09:05:34.1596901Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-12-04T09:05:34.1597918Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-12-04T09:05:34.1598862Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-12-04T09:05:34.1599826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-12-04T09:05:34.1600892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-12-04T09:05:34.1601846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-12-04T09:05:34.1602798Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-12-04T09:05:34.1603719Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-12-04T09:05:34.1621130Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-12-04T09:05:34.1699829Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-12-04T09:05:34.1701222Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:05:34.1702132Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:05:34.1702954Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-12-04T09:05:34.1703705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-12-04T09:05:34.1704444Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-12-04T09:05:34.1705168Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2025-12-04T09:05:34.1705867Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-12-04T09:05:34.1706501Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-12-04T09:05:34.1707149Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-12-04T09:05:34.1870896Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-12-04T09:05:34.1924855Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-12-04T09:05:34.1925523Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-12-04T09:05:34.1926089Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-12-04T09:05:34.1926754Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:05:34.1931700Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:05:34.1932482Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/ 2025-12-04T09:05:34.1933255Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:05:34.1934082Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:05:34.1934881Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:05:34.1935823Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:05:34.1936886Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:05:34.1937725Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:05:34.1938554Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:05:34.1939349Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:05:34.1940526Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:05:34.1941873Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:05:34.1942805Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:05:34.1944315Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:05:34.1946034Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:05:34.1946937Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-12-04T09:05:34.1948012Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-12-04T09:05:34.2004057Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-12-04T09:05:34.2060654Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-12-04T09:05:34.2062061Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-12-04T09:05:34.2119251Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-12-04T09:05:34.2120601Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-12-04T09:05:34.2121806Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-12-04T09:05:34.2123047Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-12-04T09:05:34.2124576Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-12-04T09:05:34.2125786Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-12-04T09:05:34.2126996Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-12-04T09:05:34.2128202Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-12-04T09:05:34.2129353Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-12-04T09:05:34.2130459Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-12-04T09:05:34.2131530Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-12-04T09:05:34.2132579Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-12-04T09:05:34.2133628Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-12-04T09:05:34.2134663Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-12-04T09:05:34.2135711Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-12-04T09:05:34.2209610Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-12-04T09:05:34.2210543Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-12-04T09:05:34.2286963Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-12-04T09:05:34.2288054Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:05:34.2288771Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2025-12-04T09:05:34.2289504Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-12-04T09:05:34.2290264Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-12-04T09:05:34.2291142Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-12-04T09:05:34.2292141Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-12-04T09:05:34.2293106Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-12-04T09:05:34.2293979Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-12-04T09:05:34.2294900Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-12-04T09:05:34.2295824Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-12-04T09:05:34.2296921Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-12-04T09:05:34.2297845Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-12-04T09:05:34.2298851Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-12-04T09:05:34.2312840Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-12-04T09:05:34.2375451Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-12-04T09:05:34.2376612Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:05:34.2377530Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:05:34.2378335Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-12-04T09:05:34.2379077Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-12-04T09:05:34.2379807Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-12-04T09:05:34.2380550Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2025-12-04T09:05:34.2381226Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-12-04T09:05:34.2381856Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-12-04T09:05:34.2382499Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-12-04T09:05:34.2419486Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-12-04T09:05:34.2420254Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-12-04T09:05:34.2420873Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-12-04T09:05:34.2421612Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-12-04T09:05:34.2427477Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-12-04T09:05:34.2428321Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/ 2025-12-04T09:05:34.2429171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeSystem.cmake 2025-12-04T09:05:34.2430077Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/ 2025-12-04T09:05:34.2430959Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/tmp/ 2025-12-04T09:05:34.2431946Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/CMakeCCompilerId.c 2025-12-04T09:05:34.2432966Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdC/a.out 2025-12-04T09:05:34.2433930Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCCompiler.cmake 2025-12-04T09:05:34.2434853Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/ 2025-12-04T09:05:34.2435736Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/tmp/ 2025-12-04T09:05:34.2436785Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-12-04T09:05:34.2437857Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCXX/a.out 2025-12-04T09:05:34.2438838Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCXXCompiler.cmake 2025-12-04T09:05:34.2440481Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_C.bin 2025-12-04T09:05:34.2442260Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CXX.bin 2025-12-04T09:05:34.2443423Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/ 2025-12-04T09:05:34.2444305Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/ 2025-12-04T09:05:34.2499546Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-12-04T09:05:34.2556915Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-12-04T09:05:34.2558274Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-12-04T09:05:34.2616714Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-12-04T09:05:34.2618181Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-12-04T09:05:34.2619476Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-12-04T09:05:34.2620782Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-12-04T09:05:34.2622046Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-12-04T09:05:34.2623299Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-12-04T09:05:34.2624972Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-12-04T09:05:34.2626241Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-12-04T09:05:34.2627582Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-12-04T09:05:34.2628769Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-12-04T09:05:34.2629911Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-12-04T09:05:34.2631028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-12-04T09:05:34.2632164Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-12-04T09:05:34.2633264Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/tmp/a_dlink.o 2025-12-04T09:05:34.2634385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-12-04T09:05:34.2705478Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CompilerIdCUDA/a.out 2025-12-04T09:05:34.2706485Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeCUDACompiler.cmake 2025-12-04T09:05:34.2783458Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.6/CMakeDetermineCompilerABI_CUDA.bin 2025-12-04T09:05:34.2784615Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-12-04T09:05:34.2785383Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2025-12-04T09:05:34.2786161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-12-04T09:05:34.2787001Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-12-04T09:05:34.2788260Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-12-04T09:05:34.2789577Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-12-04T09:05:34.2790634Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-12-04T09:05:34.2791754Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-12-04T09:05:34.2792790Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-12-04T09:05:34.2793836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-12-04T09:05:34.2794866Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-12-04T09:05:34.2795900Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-12-04T09:05:34.2796921Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-12-04T09:05:34.2798027Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-12-04T09:05:34.2908693Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-12-04T09:05:34.2909794Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-12-04T09:05:34.2910838Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-12-04T09:05:34.2912003Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-12-04T09:05:34.2913113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-12-04T09:05:34.2914173Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-12-04T09:05:34.2915268Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-12-04T09:05:34.2916366Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-12-04T09:05:34.2917453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-12-04T09:05:34.2918550Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-12-04T09:05:34.2919631Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-12-04T09:05:34.2934279Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-12-04T09:05:34.2987956Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-12-04T09:05:34.2989163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-12-04T09:05:34.2990165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-12-04T09:05:34.2991077Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-12-04T09:05:34.2991889Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-12-04T09:05:34.2992706Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-12-04T09:05:34.2993520Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2025-12-04T09:05:34.2994289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-12-04T09:05:34.2994976Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-12-04T09:05:34.2995883Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-12-04T09:05:34.3093911Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-12-04T09:05:34.3130925Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-12-04T09:05:34.3131516Z creating: build/lib/ 2025-12-04T09:05:34.3212679Z inflating: build/lib/libprotobuf-lite.a 2025-12-04T09:05:34.3635900Z inflating: build/lib/libprotobuf.a 2025-12-04T09:05:34.4114253Z inflating: build/lib/libprotoc.a 2025-12-04T09:05:34.4123999Z inflating: build/lib/libpthreadpool.a 2025-12-04T09:05:34.4132676Z inflating: build/lib/libcpuinfo.a 2025-12-04T09:05:34.4140530Z inflating: build/lib/libcpuinfo_internals.a 2025-12-04T09:05:34.4141319Z inflating: build/lib/libclog.a 2025-12-04T09:05:34.4162425Z inflating: build/lib/libpytorch_qnnpack.a 2025-12-04T09:05:34.4163339Z inflating: build/lib/libnnpack_reference_layers.a 2025-12-04T09:05:34.4181980Z inflating: build/lib/libnnpack.a 2025-12-04T09:05:34.4358756Z inflating: build/lib/libmicrokernels-prod.a 2025-12-04T09:05:34.5176317Z inflating: build/lib/libmicrokernels-all.a 2025-12-04T09:05:34.5244575Z inflating: build/lib/libgtest.a 2025-12-04T09:05:34.5260753Z inflating: build/lib/libgmock.a 2025-12-04T09:05:34.5261232Z inflating: build/lib/libgtest_main.a 2025-12-04T09:05:34.5261776Z inflating: build/lib/libgmock_main.a 2025-12-04T09:05:34.5349161Z inflating: build/lib/libXNNPACK.a 2025-12-04T09:05:34.5422096Z inflating: build/lib/libbenchmark.a 2025-12-04T09:05:34.5423012Z inflating: build/lib/libbenchmark_main.a 2025-12-04T09:05:34.5423466Z inflating: build/lib/libjitprofiling.a 2025-12-04T09:05:34.5430455Z inflating: build/lib/libittnotify.a 2025-12-04T09:05:34.5494670Z inflating: build/lib/libasmjit.a 2025-12-04T09:05:34.6586479Z inflating: build/lib/libfbgemm.a 2025-12-04T09:05:34.6616192Z inflating: build/lib/libtensorpipe_uv.a 2025-12-04T09:05:34.7131069Z inflating: build/lib/libtensorpipe.a 2025-12-04T09:05:34.7365436Z inflating: build/lib/libtensorpipe_cuda.a 2025-12-04T09:05:34.7493301Z inflating: build/lib/libgloo.a 2025-12-04T09:05:34.7539954Z inflating: build/lib/libonnx_proto.a 2025-12-04T09:05:34.7947980Z inflating: build/lib/libgloo_cuda.a 2025-12-04T09:05:34.8635130Z inflating: build/lib/libonnx.a 2025-12-04T09:05:34.8655093Z inflating: build/lib/libfmt.a 2025-12-04T09:05:35.8323387Z inflating: build/lib/libdnnl.a 2025-12-04T09:05:35.8777032Z inflating: build/lib/libkineto.a 2025-12-04T09:05:35.8890423Z inflating: build/lib/libc10.so 2025-12-04T09:05:35.8940067Z inflating: build/lib/libc10_cuda.so 2025-12-04T09:05:35.8940568Z inflating: build/lib/libcaffe2_nvrtc.so 2025-12-04T09:05:35.8941796Z inflating: build/lib/libtorch_global_deps.so 2025-12-04T09:05:38.8546811Z inflating: build/lib/libtorch_cpu.so 2025-12-04T09:05:38.9312057Z inflating: build/lib/libtorch_nvshmem.so 2025-12-04T09:05:41.8385507Z inflating: build/lib/libtorch_cuda.so 2025-12-04T09:05:41.8386142Z inflating: build/lib/libtorch.so 2025-12-04T09:05:41.8435712Z inflating: build/lib/libtorch_cuda_linalg.so 2025-12-04T09:05:41.8503971Z inflating: build/lib/libtorchbind_test.so 2025-12-04T09:05:41.8524421Z inflating: build/lib/libjitbackend_test.so 2025-12-04T09:05:41.8548315Z inflating: build/lib/libbackend_with_compiler.so 2025-12-04T09:05:41.8574039Z inflating: build/lib/libaoti_custom_ops.so 2025-12-04T09:05:41.8575079Z inflating: build/lib/libc10d_cuda_test.so 2025-12-04T09:05:41.8579346Z inflating: build/lib/libshm.so 2025-12-04T09:05:42.0844243Z inflating: build/lib/libtorch_python.so 2025-12-04T09:05:42.0878950Z inflating: build/lib/libnnapi_backend.so 2025-12-04T09:05:42.0879387Z creating: build/bin/ 2025-12-04T09:05:42.1317370Z inflating: build/bin/protoc-3.13.0.0 2025-12-04T09:05:42.1755816Z inflating: build/bin/protoc 2025-12-04T09:05:42.1813678Z inflating: build/bin/c10_AllocatorConfig_test 2025-12-04T09:05:42.1868066Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-12-04T09:05:42.1923825Z inflating: build/bin/c10_DeviceGuard_test 2025-12-04T09:05:42.1979429Z inflating: build/bin/c10_Device_test 2025-12-04T09:05:42.2043707Z inflating: build/bin/c10_DispatchKeySet_test 2025-12-04T09:05:42.2100851Z inflating: build/bin/c10_Scalar_test 2025-12-04T09:05:42.2154908Z inflating: build/bin/c10_StreamGuard_test 2025-12-04T09:05:42.2215718Z inflating: build/bin/c10_SymInt_test 2025-12-04T09:05:42.2272405Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-12-04T09:05:42.2332589Z inflating: build/bin/c10_InlineStreamGuard_test 2025-12-04T09:05:42.2387454Z inflating: build/bin/c10_ConstexprCrc_test 2025-12-04T09:05:42.2447370Z inflating: build/bin/c10_SizesAndStrides_test 2025-12-04T09:05:42.2519054Z inflating: build/bin/c10_cow_test 2025-12-04T09:05:42.2577361Z inflating: build/bin/c10_Bitset_test 2025-12-04T09:05:42.2629174Z inflating: build/bin/c10_ArrayRef_test 2025-12-04T09:05:42.2683455Z inflating: build/bin/c10_DeadlockDetection_test 2025-12-04T09:05:42.2741128Z inflating: build/bin/c10_IntrusiveList_test 2025-12-04T09:05:42.2799472Z inflating: build/bin/c10_LeftRight_test 2025-12-04T09:05:42.2855514Z inflating: build/bin/c10_Half_test 2025-12-04T09:05:42.2908660Z inflating: build/bin/c10_Semaphore_test 2025-12-04T09:05:42.2970995Z inflating: build/bin/c10_Enumerate_test 2025-12-04T09:05:42.3027138Z inflating: build/bin/c10_NetworkFlow_test 2025-12-04T09:05:42.3082963Z inflating: build/bin/c10_Synchronized_test 2025-12-04T09:05:42.3142650Z inflating: build/bin/c10_ThreadLocal_test 2025-12-04T09:05:42.3196482Z inflating: build/bin/c10_accumulate_test 2025-12-04T09:05:42.3253653Z inflating: build/bin/c10_TypeIndex_test 2025-12-04T09:05:42.3307084Z inflating: build/bin/c10_bit_cast_test 2025-12-04T09:05:42.3368760Z inflating: build/bin/c10_bfloat16_test 2025-12-04T09:05:42.3429056Z inflating: build/bin/c10_complex_math_test 2025-12-04T09:05:42.3486031Z inflating: build/bin/c10_exception_test 2025-12-04T09:05:42.3540259Z inflating: build/bin/c10_error_test 2025-12-04T09:05:42.3597576Z inflating: build/bin/c10_complex_test 2025-12-04T09:05:42.3652944Z inflating: build/bin/c10_flags_test 2025-12-04T09:05:42.3706922Z inflating: build/bin/c10_generic_math_test 2025-12-04T09:05:42.3864300Z inflating: build/bin/c10_intrusive_ptr_test 2025-12-04T09:05:42.3918836Z inflating: build/bin/c10_irange_test 2025-12-04T09:05:42.3977823Z inflating: build/bin/c10_lazy_test 2025-12-04T09:05:42.4030193Z inflating: build/bin/c10_nofatal_test 2025-12-04T09:05:42.4091535Z inflating: build/bin/c10_logging_test 2025-12-04T09:05:42.4171019Z inflating: build/bin/c10_optional_test 2025-12-04T09:05:42.4235464Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-12-04T09:05:42.4389363Z inflating: build/bin/c10_small_vector_test 2025-12-04T09:05:42.4448126Z inflating: build/bin/c10_registry_test 2025-12-04T09:05:42.4507757Z inflating: build/bin/c10_string_util_test 2025-12-04T09:05:42.4565133Z inflating: build/bin/c10_ssize_test 2025-12-04T09:05:42.4618075Z inflating: build/bin/c10_string_view_test 2025-12-04T09:05:42.4665017Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-12-04T09:05:42.4717474Z inflating: build/bin/c10_tempfile_test 2025-12-04T09:05:42.4778684Z inflating: build/bin/c10_typeid_test 2025-12-04T09:05:42.4834360Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2025-12-04T09:05:42.4893535Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2025-12-04T09:05:42.4951045Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2025-12-04T09:05:42.5005912Z inflating: build/bin/c10_cuda_CUDATest 2025-12-04T09:05:42.5062574Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2025-12-04T09:05:42.5117432Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2025-12-04T09:05:42.5176211Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2025-12-04T09:05:42.5231545Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2025-12-04T09:05:42.5802513Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-12-04T09:05:42.6389320Z inflating: build/bin/vec_test_all_types_AVX512 2025-12-04T09:05:42.6988659Z inflating: build/bin/vec_test_all_types_AVX2 2025-12-04T09:05:42.7042447Z inflating: build/bin/test_vec_half_DEFAULT 2025-12-04T09:05:42.7143873Z inflating: build/bin/test_aoti_abi_check 2025-12-04T09:05:42.7196262Z inflating: build/bin/test_vec_half_AVX512 2025-12-04T09:05:42.7251080Z inflating: build/bin/test_vec_half_AVX2 2025-12-04T09:05:42.7326292Z inflating: build/bin/Dict_test 2025-12-04T09:05:42.7383417Z inflating: build/bin/Dimname_test 2025-12-04T09:05:42.7453367Z inflating: build/bin/MaybeOwned_test 2025-12-04T09:05:42.7511999Z inflating: build/bin/NamedTensor_test 2025-12-04T09:05:42.7576970Z inflating: build/bin/apply_utils_test 2025-12-04T09:05:42.7638751Z inflating: build/bin/atest 2025-12-04T09:05:42.7705879Z inflating: build/bin/basic 2025-12-04T09:05:42.7765805Z inflating: build/bin/broadcast_test 2025-12-04T09:05:42.7819350Z inflating: build/bin/cpu_allocator_test 2025-12-04T09:05:42.7882321Z inflating: build/bin/cpu_generator_test 2025-12-04T09:05:42.7939825Z inflating: build/bin/cpu_profiling_allocator_test 2025-12-04T09:05:42.8034429Z inflating: build/bin/cpu_rng_test 2025-12-04T09:05:42.8090328Z inflating: build/bin/dlconvertor_test 2025-12-04T09:05:42.8152709Z inflating: build/bin/extension_backend_test 2025-12-04T09:05:42.8212487Z inflating: build/bin/half_test 2025-12-04T09:05:42.8313310Z inflating: build/bin/ivalue_test 2025-12-04T09:05:42.8369700Z inflating: build/bin/lazy_tensor_test 2025-12-04T09:05:42.8425217Z inflating: build/bin/math_kernel_test 2025-12-04T09:05:42.8482095Z inflating: build/bin/memory_format_test 2025-12-04T09:05:42.8540171Z inflating: build/bin/memory_overlapping_test 2025-12-04T09:05:42.8595322Z inflating: build/bin/mobile_memory_cleanup 2025-12-04T09:05:42.8656191Z inflating: build/bin/native_test 2025-12-04T09:05:42.8709020Z inflating: build/bin/operator_name_test 2025-12-04T09:05:42.8766317Z inflating: build/bin/operators_test 2025-12-04T09:05:42.8821264Z inflating: build/bin/packedtensoraccessor_test 2025-12-04T09:05:42.8892974Z inflating: build/bin/pow_test 2025-12-04T09:05:42.8953951Z inflating: build/bin/quantized_test 2025-12-04T09:05:42.9008819Z inflating: build/bin/reduce_ops_test 2025-12-04T09:05:42.9063478Z inflating: build/bin/reportMemoryUsage_test 2025-12-04T09:05:42.9122692Z inflating: build/bin/scalar_tensor_test 2025-12-04T09:05:42.9183583Z inflating: build/bin/scalar_test 2025-12-04T09:05:42.9238351Z inflating: build/bin/StorageUtils_test 2025-12-04T09:05:42.9294664Z inflating: build/bin/stride_properties_test 2025-12-04T09:05:42.9380069Z inflating: build/bin/tensor_iterator_test 2025-12-04T09:05:42.9437020Z inflating: build/bin/test_parallel 2025-12-04T09:05:42.9492219Z inflating: build/bin/thread_init_test 2025-12-04T09:05:42.9550771Z inflating: build/bin/type_ptr_test 2025-12-04T09:05:42.9614638Z inflating: build/bin/type_test 2025-12-04T09:05:42.9668915Z inflating: build/bin/undefined_tensor_test 2025-12-04T09:05:42.9723006Z inflating: build/bin/verify_api_visibility 2025-12-04T09:05:42.9796271Z inflating: build/bin/legacy_vmap_test 2025-12-04T09:05:42.9852499Z inflating: build/bin/weakref_test 2025-12-04T09:05:42.9908196Z inflating: build/bin/wrapdim_test 2025-12-04T09:05:42.9965709Z inflating: build/bin/xla_tensor_test 2025-12-04T09:05:43.0027380Z inflating: build/bin/IListRef_test 2025-12-04T09:05:43.0134568Z inflating: build/bin/List_test 2025-12-04T09:05:43.0205396Z inflating: build/bin/KernelFunction_test 2025-12-04T09:05:43.0325530Z inflating: build/bin/kernel_function_legacy_test 2025-12-04T09:05:43.0423473Z inflating: build/bin/kernel_function_test 2025-12-04T09:05:43.0552199Z inflating: build/bin/kernel_lambda_legacy_test 2025-12-04T09:05:43.0656449Z inflating: build/bin/kernel_lambda_test 2025-12-04T09:05:43.0718527Z inflating: build/bin/kernel_stackbased_test 2025-12-04T09:05:43.0817773Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-12-04T09:05:43.0872378Z inflating: build/bin/CppSignature_test 2025-12-04T09:05:43.0931410Z inflating: build/bin/backend_fallback_test 2025-12-04T09:05:43.0986010Z inflating: build/bin/op_allowlist_test 2025-12-04T09:05:43.1294372Z inflating: build/bin/op_registration_test 2025-12-04T09:05:43.1366561Z inflating: build/bin/inline_container_test 2025-12-04T09:05:43.1422600Z inflating: build/bin/cuda_allocator_test 2025-12-04T09:05:43.1479190Z inflating: build/bin/cuda_apply_test 2025-12-04T09:05:43.1545079Z inflating: build/bin/cuda_atomic_ops_test 2025-12-04T09:05:43.1604792Z inflating: build/bin/cuda_caching_host_allocator_test 2025-12-04T09:05:43.1677073Z inflating: build/bin/cuda_complex_math_test 2025-12-04T09:05:43.1741860Z inflating: build/bin/cuda_complex_test 2025-12-04T09:05:43.1809386Z inflating: build/bin/cuda_cub_test 2025-12-04T09:05:43.1866068Z inflating: build/bin/cuda_cublas_handle_pool_test 2025-12-04T09:05:43.1918066Z inflating: build/bin/cuda_device_test 2025-12-04T09:05:43.1998761Z inflating: build/bin/cuda_distributions_test 2025-12-04T09:05:43.2055583Z inflating: build/bin/cuda_dlconvertor_test 2025-12-04T09:05:43.2111982Z inflating: build/bin/cuda_event_test 2025-12-04T09:05:43.2168313Z inflating: build/bin/cuda_exchange_device_test 2025-12-04T09:05:43.2227452Z inflating: build/bin/cuda_generator_test 2025-12-04T09:05:43.2282377Z inflating: build/bin/cuda_half_test 2025-12-04T09:05:43.2335337Z inflating: build/bin/cuda_allocatorTraceTracker_test 2025-12-04T09:05:43.2401878Z inflating: build/bin/cuda_stream_test 2025-12-04T09:05:43.2457833Z inflating: build/bin/cuda_reportMemoryUsage_test 2025-12-04T09:05:43.2509545Z inflating: build/bin/cuda_cudnn_test 2025-12-04T09:05:43.2567849Z inflating: build/bin/cuda_integer_divider_test 2025-12-04T09:05:43.2620454Z inflating: build/bin/cuda_optional_test 2025-12-04T09:05:43.2675124Z inflating: build/bin/cuda_packedtensoraccessor_test 2025-12-04T09:05:43.2731702Z inflating: build/bin/cuda_vectorized_test 2025-12-04T09:05:43.3811059Z inflating: build/bin/test_jit 2025-12-04T09:05:43.4159777Z inflating: build/bin/test_lazy 2025-12-04T09:05:43.4216425Z inflating: build/bin/BackoffTest 2025-12-04T09:05:43.4271611Z inflating: build/bin/FileStoreTest 2025-12-04T09:05:43.4332548Z inflating: build/bin/TCPStoreTest 2025-12-04T09:05:43.4390563Z inflating: build/bin/HashStoreTest 2025-12-04T09:05:43.4406658Z inflating: build/bin/ProcessGroupMPITest 2025-12-04T09:05:43.4407981Z inflating: build/bin/example_allreduce 2025-12-04T09:05:43.4468001Z inflating: build/bin/test_dist_autograd 2025-12-04T09:05:43.4541652Z inflating: build/bin/test_cpp_rpc 2025-12-04T09:05:43.4612567Z inflating: build/bin/ProcessGroupGlooTest 2025-12-04T09:05:43.4671330Z inflating: build/bin/ProcessGroupGlooAsyncTest 2025-12-04T09:05:43.4740975Z inflating: build/bin/ProcessGroupNCCLTest 2025-12-04T09:05:43.4806781Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2025-12-04T09:05:43.5953327Z inflating: build/bin/test_api 2025-12-04T09:05:43.5955704Z inflating: build/bin/parallel_benchmark 2025-12-04T09:05:43.5960219Z inflating: build/bin/torch_shm_manager 2025-12-04T09:05:43.5960609Z creating: .additional_ci_files/ 2025-12-04T09:05:43.6024374Z inflating: .additional_ci_files/test-times.json 2025-12-04T09:05:43.6254361Z inflating: .additional_ci_files/test-class-times.json 2025-12-04T09:05:43.6284026Z ##[group]Run rm artifacts.zip 2025-12-04T09:05:43.6284339Z rm artifacts.zip 2025-12-04T09:05:43.6292724Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:43.6293126Z env: 2025-12-04T09:05:43.6293494Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:43.6293762Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:43.6294094Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:43.6294470Z ##[endgroup] 2025-12-04T09:05:43.6993843Z ##[group]Run df -H 2025-12-04T09:05:43.6994123Z df -H 2025-12-04T09:05:43.7000235Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:43.7000649Z env: 2025-12-04T09:05:43.7000872Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:43.7001164Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:43.7001505Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:43.7001897Z ##[endgroup] 2025-12-04T09:05:43.7046781Z Filesystem Size Used Avail Use% Mounted on 2025-12-04T09:05:43.7047356Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-12-04T09:05:43.7047756Z tmpfs 101G 0 101G 0% /dev/shm 2025-12-04T09:05:43.7048150Z tmpfs 41G 693k 41G 1% /run 2025-12-04T09:05:43.7048590Z /dev/nvme0n1p1 161G 54G 108G 34% / 2025-12-04T09:05:43.7048967Z tmpfs 101G 17k 101G 1% /tmp 2025-12-04T09:05:43.7049357Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-12-04T09:05:43.7049751Z tmpfs 21G 0 21G 0% /run/user/0 2025-12-04T09:05:43.7086144Z Prepare all required actions 2025-12-04T09:05:43.7086789Z Getting action download info 2025-12-04T09:05:43.9103479Z ##[group]Run ./.github/actions/download-td-artifacts 2025-12-04T09:05:43.9103843Z with: 2025-12-04T09:05:43.9104051Z env: 2025-12-04T09:05:43.9104256Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:43.9104700Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:43.9105054Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:43.9105436Z ##[endgroup] 2025-12-04T09:05:43.9148443Z ##[group]Run seemethere/download-artifact-s3@v4 2025-12-04T09:05:43.9148843Z with: 2025-12-04T09:05:43.9149085Z name: td_results 2025-12-04T09:05:43.9149350Z s3-bucket: gha-artifacts 2025-12-04T09:05:43.9149657Z region: us-east-1 2025-12-04T09:05:43.9149916Z env: 2025-12-04T09:05:43.9150149Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:43.9150456Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:43.9150822Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:43.9151289Z ##[endgroup] 2025-12-04T09:05:44.3949285Z (node:62824) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-12-04T09:05:44.3949878Z 2025-12-04T09:05:44.3950111Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-12-04T09:05:44.3950735Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-12-04T09:05:44.3951430Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-12-04T09:05:44.4957232Z Found 1 objects with prefix pytorch/pytorch/19922768520/td_results/ 2025-12-04T09:05:44.4957987Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2025-12-04T09:05:44.5764318Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2025-12-04T09:05:44.5768483Z Artifact download has finished successfully 2025-12-04T09:05:44.5939881Z ##[group]Run mkdir -p .additional_ci_files 2025-12-04T09:05:44.5940321Z mkdir -p .additional_ci_files 2025-12-04T09:05:44.5940789Z mv td_results.json .additional_ci_files/td_results.json || true 2025-12-04T09:05:44.5948274Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:44.5948715Z env: 2025-12-04T09:05:44.5948952Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:44.5949259Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:44.5949623Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:44.5950187Z ##[endgroup] 2025-12-04T09:05:44.6050962Z ##[group]Run .github/scripts/parse_ref.py 2025-12-04T09:05:44.6051359Z .github/scripts/parse_ref.py 2025-12-04T09:05:44.6056710Z shell: /usr/bin/bash -e {0} 2025-12-04T09:05:44.6056980Z env: 2025-12-04T09:05:44.6057209Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:44.6057489Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:44.6057807Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:44.6058185Z ##[endgroup] 2025-12-04T09:05:44.6276322Z Setting output branch=main 2025-12-04T09:05:44.6414727Z Prepare all required actions 2025-12-04T09:05:44.6415128Z Getting action download info 2025-12-04T09:05:44.7976830Z ##[group]Run ./.github/actions/filter-test-configs 2025-12-04T09:05:44.7977187Z with: 2025-12-04T09:05:44.7977584Z github-token: *** 2025-12-04T09:05:44.7988441Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}]} 2025-12-04T09:05:44.8000941Z job-name: linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T09:05:44.8001707Z env: 2025-12-04T09:05:44.8001925Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:44.8002204Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:44.8002528Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:44.8002886Z ##[endgroup] 2025-12-04T09:05:44.8040562Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T09:05:44.8040897Z with: 2025-12-04T09:05:44.8041123Z shell: bash 2025-12-04T09:05:44.8041355Z timeout_minutes: 10 2025-12-04T09:05:44.8041624Z max_attempts: 5 2025-12-04T09:05:44.8041882Z retry_wait_seconds: 30 2025-12-04T09:05:44.8042923Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T09:05:44.8043848Z polling_interval_seconds: 1 2025-12-04T09:05:44.8044164Z warning_on_retry: true 2025-12-04T09:05:44.8044456Z continue_on_error: false 2025-12-04T09:05:44.8044723Z env: 2025-12-04T09:05:44.8044949Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:44.8045246Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:44.8045575Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:44.8046168Z GITHUB_TOKEN: *** 2025-12-04T09:05:44.8046428Z ##[endgroup] 2025-12-04T09:05:44.9042825Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-12-04T09:05:45.1531942Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T09:05:45.2725602Z Collecting requests==2.27.1 2025-12-04T09:05:45.2888392Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-12-04T09:05:45.4753051Z Collecting pyyaml==6.0.2 2025-12-04T09:05:45.4789117Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-12-04T09:05:45.5054467Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-12-04T09:05:45.9225189Z Collecting charset-normalizer~=2.0.0 2025-12-04T09:05:45.9265926Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-12-04T09:05:45.9316214Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-12-04T09:05:45.9845093Z Collecting certifi>=2017.4.17 2025-12-04T09:05:45.9885076Z Downloading certifi-2025.11.12-py3-none-any.whl (159 kB) 2025-12-04T09:05:46.0828303Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-12-04T09:05:46.2126570Z Successfully installed certifi-2025.11.12 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-12-04T09:05:46.8859452Z Command completed after 1 attempt(s). 2025-12-04T09:05:46.8907162Z ##[group]Run set -x 2025-12-04T09:05:46.8907557Z set -x 2025-12-04T09:05:46.8908000Z  2025-12-04T09:05:46.8908444Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T09:05:46.8908994Z # in runner workspace 2025-12-04T09:05:46.8909449Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-12-04T09:05:46.8916075Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:46.8916517Z env: 2025-12-04T09:05:46.8916756Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:46.8917065Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:46.8917430Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:46.8917832Z ##[endgroup] 2025-12-04T09:05:46.8944281Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-12-04T09:05:46.9132034Z Setting output branch=main 2025-12-04T09:05:46.9187112Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T09:05:46.9187854Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-12-04T09:05:46.9188252Z echo "Job name: ${JOB_NAME}" 2025-12-04T09:05:46.9188593Z  2025-12-04T09:05:46.9189036Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-12-04T09:05:46.9189626Z # in runner workspace 2025-12-04T09:05:46.9190293Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-12-04T09:05:46.9190844Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-12-04T09:05:46.9191235Z  --job-name "${JOB_NAME}" \ 2025-12-04T09:05:46.9203134Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}]}" \ 2025-12-04T09:05:46.9213914Z  --selected-test-configs "" \ 2025-12-04T09:05:46.9214264Z  --pr-number "${PR_NUMBER}" \ 2025-12-04T09:05:46.9214575Z  --tag "${TAG}" \ 2025-12-04T09:05:46.9214885Z  --event-name "${EVENT_NAME}" \ 2025-12-04T09:05:46.9215220Z  --schedule "${SCHEDULE}" \ 2025-12-04T09:05:46.9215548Z  --branch "${HEAD_BRANCH}" 2025-12-04T09:05:46.9220987Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:46.9221383Z env: 2025-12-04T09:05:46.9221609Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:46.9221875Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:46.9222202Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:46.9222779Z GITHUB_TOKEN: *** 2025-12-04T09:05:46.9223456Z JOB_NAME: linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T09:05:46.9224471Z PR_NUMBER: 2025-12-04T09:05:46.9224725Z TAG: 2025-12-04T09:05:46.9225037Z EVENT_NAME: schedule 2025-12-04T09:05:46.9225311Z SCHEDULE: 29 8 * * * 2025-12-04T09:05:46.9225591Z HEAD_BRANCH: main 2025-12-04T09:05:46.9225856Z ##[endgroup] 2025-12-04T09:05:46.9250050Z Workflow: trunk 2025-12-04T09:05:46.9250974Z Job name: linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T09:05:47.1262844Z Setting output keep-going=True 2025-12-04T09:05:47.1263912Z Setting output ci-verbose-test-logs=False 2025-12-04T09:05:47.1264365Z Setting output ci-test-showlocals=False 2025-12-04T09:05:47.1264758Z Setting output ci-no-test-timeout=False 2025-12-04T09:05:47.1265138Z Setting output ci-no-td=False 2025-12-04T09:05:47.1265483Z Setting output ci-td-distributed=False 2025-12-04T09:05:47.1265867Z Setting output is-unstable=False 2025-12-04T09:05:47.1266215Z Setting output reenabled-issues= 2025-12-04T09:05:47.1292254Z Setting output test-matrix={"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}]} 2025-12-04T09:05:47.1317583Z Setting output is-test-matrix-empty=False 2025-12-04T09:05:47.1393081Z ##[group]Run echo "Filtered matrix:" 2025-12-04T09:05:47.1393679Z echo "Filtered matrix:" 2025-12-04T09:05:47.1418665Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 1, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 2, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 3, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 4, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "default", "shard": 5, "num_shards": 5, "runner": "lf.linux.g6.4xlarge.experimental.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 1, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "lf.linux.g4dn.12xlarge.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "pr_time_benchmarks", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "mem_leak_check": "mem_leak_check", "rerun_disabled_tests": "rerun_disabled_tests"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests", "mem_leak_check": "mem_leak_check"}, {"config": "libtorch_agnostic_targetting", "shard": 1, "num_shards": 1, "runner": "linux.g4dn.metal.nvidia.gpu", "rerun_disabled_tests": "rerun_disabled_tests"}]}" 2025-12-04T09:05:47.1444985Z  2025-12-04T09:05:47.1445204Z echo 2025-12-04T09:05:47.1445490Z echo "Is the current job unstable? False" 2025-12-04T09:05:47.1445826Z  2025-12-04T09:05:47.1446039Z echo 2025-12-04T09:05:47.1446306Z echo "Is keep-going label set? True" 2025-12-04T09:05:47.1446626Z  2025-12-04T09:05:47.1446833Z echo 2025-12-04T09:05:47.1447075Z echo "Reenabled issues? " 2025-12-04T09:05:47.1452630Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:47.1453026Z env: 2025-12-04T09:05:47.1453253Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:47.1453685Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:47.1454151Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:47.1454621Z ##[endgroup] 2025-12-04T09:05:47.1492103Z Filtered matrix: 2025-12-04T09:05:47.1524719Z {include: [{config: default, shard: 1, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: default, shard: 1, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 1, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 1, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 2, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: default, shard: 2, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 2, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 2, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 3, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: default, shard: 3, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 3, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 3, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 4, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: default, shard: 4, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 4, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 4, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 5, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: default, shard: 5, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: default, shard: 5, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: default, shard: 5, num_shards: 5, runner: lf.linux.g6.4xlarge.experimental.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 1, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: distributed, shard: 1, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 1, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: distributed, shard: 1, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 2, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: distributed, shard: 2, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 2, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: distributed, shard: 2, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 3, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: distributed, shard: 3, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: distributed, shard: 3, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: distributed, shard: 3, num_shards: 3, runner: lf.linux.g4dn.12xlarge.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: pr_time_benchmarks, shard: 1, num_shards: 1, runner: linux.g4dn.metal.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: pr_time_benchmarks, shard: 1, num_shards: 1, runner: linux.g4dn.metal.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: pr_time_benchmarks, shard: 1, num_shards: 1, runner: linux.g4dn.metal.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: pr_time_benchmarks, shard: 1, num_shards: 1, runner: linux.g4dn.metal.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}, {config: libtorch_agnostic_targetting, shard: 1, num_shards: 1, runner: linux.g4dn.metal.nvidia.gpu, mem_leak_check: mem_leak_check}, {config: libtorch_agnostic_targetting, shard: 1, num_shards: 1, runner: linux.g4dn.metal.nvidia.gpu, mem_leak_check: mem_leak_check, rerun_disabled_tests: rerun_disabled_tests}, {config: libtorch_agnostic_targetting, shard: 1, num_shards: 1, runner: linux.g4dn.metal.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests, mem_leak_check: mem_leak_check}, {config: libtorch_agnostic_targetting, shard: 1, num_shards: 1, runner: linux.g4dn.metal.nvidia.gpu, rerun_disabled_tests: rerun_disabled_tests}]} 2025-12-04T09:05:47.1550372Z 2025-12-04T09:05:47.1550503Z Is the current job unstable? False 2025-12-04T09:05:47.1550742Z 2025-12-04T09:05:47.1550878Z Is keep-going label set? True 2025-12-04T09:05:47.1551093Z 2025-12-04T09:05:47.1551214Z Reenabled issues? 2025-12-04T09:05:47.1582926Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T09:05:47.1583457Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-12-04T09:05:47.1589673Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:47.1590115Z env: 2025-12-04T09:05:47.1590368Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:47.1590666Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:47.1591029Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:47.1591454Z JOB_TIMEOUT: 600 2025-12-04T09:05:47.1591720Z ##[endgroup] 2025-12-04T09:05:47.1642063Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:05:47.1642601Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:05:47.1643071Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-12-04T09:05:47.1648210Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T09:05:47.1648602Z env: 2025-12-04T09:05:47.1648825Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:47.1649091Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:47.1649417Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:47.1649796Z ##[endgroup] 2025-12-04T09:05:47.1751710Z ##[group]Run set -x 2025-12-04T09:05:47.1752081Z set -x 2025-12-04T09:05:47.1752335Z  2025-12-04T09:05:47.1752625Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-12-04T09:05:47.1753073Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-12-04T09:05:47.1753542Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-12-04T09:05:47.1753965Z  TEST_COMMAND=.ci/onnx/test.sh 2025-12-04T09:05:47.1754303Z else 2025-12-04T09:05:47.1754593Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T09:05:47.1754952Z fi 2025-12-04T09:05:47.1755185Z  2025-12-04T09:05:47.1755477Z # Leaving 1GB for the runner and other things 2025-12-04T09:05:47.1756156Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-12-04T09:05:47.1757183Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-12-04T09:05:47.1758124Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-12-04T09:05:47.1758745Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-12-04T09:05:47.1759238Z  2025-12-04T09:05:47.1759636Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T09:05:47.1760152Z  SHM_OPTS= 2025-12-04T09:05:47.1760414Z  JENKINS_USER= 2025-12-04T09:05:47.1760779Z  # ensure that docker container cleanly exits in 12 hours 2025-12-04T09:05:47.1761268Z  # if for some reason cleanup action doesn't stop container 2025-12-04T09:05:47.1761681Z  # when job is cancelled 2025-12-04T09:05:47.1762003Z  DOCKER_SHELL_CMD="sleep 12h" 2025-12-04T09:05:47.1762338Z  USED_IMAGE="${DOCKER_IMAGE_S390X}" 2025-12-04T09:05:47.1762665Z else 2025-12-04T09:05:47.1762927Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-12-04T09:05:47.1763273Z  JENKINS_USER="--user jenkins" 2025-12-04T09:05:47.1763595Z  DOCKER_SHELL_CMD= 2025-12-04T09:05:47.1763891Z  USED_IMAGE="${DOCKER_IMAGE}" 2025-12-04T09:05:47.1764197Z fi 2025-12-04T09:05:47.1764396Z  2025-12-04T09:05:47.1764741Z # detached container should get cleaned up by teardown_ec2_linux 2025-12-04T09:05:47.1765295Z # TODO: Stop building test binaries as part of the build phase 2025-12-04T09:05:47.1765919Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-12-04T09:05:47.1766479Z # shellcheck disable=SC2086,SC2090 2025-12-04T09:05:47.1766831Z container_name=$(docker run \ 2025-12-04T09:05:47.1767151Z  ${GPU_FLAG:-} \ 2025-12-04T09:05:47.1767450Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-12-04T09:05:47.1767813Z  -e BUILD_ENVIRONMENT \ 2025-12-04T09:05:47.1768126Z  -e PR_NUMBER \ 2025-12-04T09:05:47.1768398Z  -e GITHUB_ACTIONS \ 2025-12-04T09:05:47.1768697Z  -e GITHUB_REPOSITORY \ 2025-12-04T09:05:47.1769004Z  -e GITHUB_WORKFLOW \ 2025-12-04T09:05:47.1769287Z  -e GITHUB_JOB \ 2025-12-04T09:05:47.1769566Z  -e GITHUB_RUN_ID \ 2025-12-04T09:05:47.1769857Z  -e GITHUB_RUN_NUMBER \ 2025-12-04T09:05:47.1770160Z  -e GITHUB_RUN_ATTEMPT \ 2025-12-04T09:05:47.1770452Z  -e JOB_ID \ 2025-12-04T09:05:47.1770710Z  -e JOB_NAME \ 2025-12-04T09:05:47.1770977Z  -e BASE_SHA \ 2025-12-04T09:05:47.1771229Z  -e BRANCH \ 2025-12-04T09:05:47.1771484Z  -e SHA1 \ 2025-12-04T09:05:47.1771744Z  -e AWS_DEFAULT_REGION \ 2025-12-04T09:05:47.1772042Z  -e IN_WHEEL_TEST \ 2025-12-04T09:05:47.1772328Z  -e SHARD_NUMBER \ 2025-12-04T09:05:47.1772609Z  -e TEST_CONFIG \ 2025-12-04T09:05:47.1772887Z  -e NUM_TEST_SHARDS \ 2025-12-04T09:05:47.1773288Z  -e REENABLED_ISSUES \ 2025-12-04T09:05:47.1773607Z  -e CONTINUE_THROUGH_ERROR \ 2025-12-04T09:05:47.1773935Z  -e VERBOSE_TEST_LOGS \ 2025-12-04T09:05:47.1774231Z  -e TEST_SHOWLOCALS \ 2025-12-04T09:05:47.1774529Z  -e NO_TEST_TIMEOUT \ 2025-12-04T09:05:47.1774816Z  -e NO_TD \ 2025-12-04T09:05:47.1775063Z  -e TD_DISTRIBUTED \ 2025-12-04T09:05:47.1775348Z  -e PR_LABELS \ 2025-12-04T09:05:47.1775647Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-12-04T09:05:47.1775980Z  -e SCCACHE_BUCKET \ 2025-12-04T09:05:47.1776272Z  -e SCCACHE_REGION \ 2025-12-04T09:05:47.1776557Z  -e XLA_CUDA \ 2025-12-04T09:05:47.1776838Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-12-04T09:05:47.1777209Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-12-04T09:05:47.1777590Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-12-04T09:05:47.1778033Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-12-04T09:05:47.1778371Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-12-04T09:05:47.1778714Z  -e VLLM_TEST_HUGGING_FACE_TOKEN \ 2025-12-04T09:05:47.1779070Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-12-04T09:05:47.1779396Z  -e DASHBOARD_TAG \ 2025-12-04T09:05:47.1779693Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-12-04T09:05:47.1780064Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-12-04T09:05:47.1780496Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-12-04T09:05:47.1780914Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-12-04T09:05:47.1781324Z  --security-opt seccomp=unconfined \ 2025-12-04T09:05:47.1781680Z  --cap-add=SYS_PTRACE \ 2025-12-04T09:05:47.1781976Z  --ipc=host \ 2025-12-04T09:05:47.1782237Z  ${SHM_OPTS} \ 2025-12-04T09:05:47.1782493Z  --tty \ 2025-12-04T09:05:47.1782729Z  --detach \ 2025-12-04T09:05:47.1783002Z  --name="${container_name}" \ 2025-12-04T09:05:47.1783323Z  ${JENKINS_USER} \ 2025-12-04T09:05:47.1783675Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-12-04T09:05:47.1784075Z  -w /var/lib/jenkins/workspace \ 2025-12-04T09:05:47.1784399Z  "${USED_IMAGE}" \ 2025-12-04T09:05:47.1784678Z  ${DOCKER_SHELL_CMD} 2025-12-04T09:05:47.1784937Z ) 2025-12-04T09:05:47.1785277Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-12-04T09:05:47.1785697Z  2025-12-04T09:05:47.1785958Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-12-04T09:05:47.1786560Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-12-04T09:05:47.1787103Z fi 2025-12-04T09:05:47.1787419Z  2025-12-04T09:05:47.1788128Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-12-04T09:05:47.1794087Z shell: /usr/bin/bash -e {0} 2025-12-04T09:05:47.1794398Z env: 2025-12-04T09:05:47.1794628Z GIT_DEFAULT_BRANCH: main 2025-12-04T09:05:47.1794932Z HAS_NVIDIA_GPU: true 2025-12-04T09:05:47.1795293Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:05:47.1795782Z BUILD_ENVIRONMENT: linux-jammy-cuda12.8-py3.10-gcc11 2025-12-04T09:05:47.1796197Z PR_NUMBER: 2025-12-04T09:05:47.1796474Z GITHUB_REPOSITORY: pytorch/pytorch 2025-12-04T09:05:47.1796831Z GITHUB_WORKFLOW: trunk 2025-12-04T09:05:47.1797109Z GITHUB_JOB: test 2025-12-04T09:05:47.1797374Z GITHUB_RUN_ID: 19922768520 2025-12-04T09:05:47.1797682Z GITHUB_RUN_NUMBER: 158165 2025-12-04T09:05:47.1797978Z GITHUB_RUN_ATTEMPT: 1 2025-12-04T09:05:47.1798256Z JOB_ID: 57116084912 2025-12-04T09:05:47.1798909Z JOB_NAME: linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T09:05:47.1799832Z BRANCH: main 2025-12-04T09:05:47.1800251Z SHA1: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:05:47.1800654Z BASE_SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:05:47.1801020Z TEST_CONFIG: distributed 2025-12-04T09:05:47.1801273Z SHARD_NUMBER: 2 2025-12-04T09:05:47.1801505Z NUM_TEST_SHARDS: 3 2025-12-04T09:05:47.1801741Z EXTRA_FLAGS: 2025-12-04T09:05:47.1801961Z OP_BENCHMARK_TESTS: 2025-12-04T09:05:47.1802217Z REENABLED_ISSUES: 2025-12-04T09:05:47.1802473Z CONTINUE_THROUGH_ERROR: True 2025-12-04T09:05:47.1802750Z VERBOSE_TEST_LOGS: False 2025-12-04T09:05:47.1803025Z TEST_SHOWLOCALS: False 2025-12-04T09:05:47.1803291Z NO_TEST_TIMEOUT: False 2025-12-04T09:05:47.1803549Z NO_TD: False 2025-12-04T09:05:47.1803771Z TD_DISTRIBUTED: False 2025-12-04T09:05:47.1804094Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-12-04T09:05:47.1804473Z SCCACHE_REGION: us-east-1 2025-12-04T09:05:47.1804806Z SHM_SIZE: 2g 2025-12-04T09:05:47.1805617Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:05:47.1807093Z DOCKER_IMAGE_S390X: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:05:47.1807996Z XLA_CUDA: 2025-12-04T09:05:47.1808369Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:05:47.1808835Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 1 2025-12-04T09:05:47.1809173Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-12-04T09:05:47.1809490Z DASHBOARD_TAG: 2025-12-04T09:05:47.1809944Z VLLM_TEST_HUGGING_FACE_TOKEN: *** 2025-12-04T09:05:47.1810357Z HUGGING_FACE_HUB_TOKEN: *** 2025-12-04T09:05:47.1810788Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-12-04T09:05:47.1811336Z ARTIFACTS_FILE_SUFFIX: test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912 2025-12-04T09:05:47.1811891Z ##[endgroup] 2025-12-04T09:05:47.1834623Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2025-12-04T09:05:47.1835096Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *onnx* ]] 2025-12-04T09:05:47.1835516Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-12-04T09:05:47.1837720Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-12-04T09:05:47.1858693Z + TOTAL_AVAILABLE_MEMORY_IN_GB='185.682 ' 2025-12-04T09:05:47.1859114Z + TOTAL_MEMORY_WITH_SWAP=188 2025-12-04T09:05:47.1859513Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *\s\3\9\0\x* ]] 2025-12-04T09:05:47.1859941Z + SHM_OPTS=--shm-size=2g 2025-12-04T09:05:47.1860250Z + JENKINS_USER='--user jenkins' 2025-12-04T09:05:47.1860550Z + DOCKER_SHELL_CMD= 2025-12-04T09:05:47.1861442Z + USED_IMAGE=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:05:47.1867156Z +++ nproc --ignore=2 2025-12-04T09:05:47.1902457Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=185g --memory-swap=188g --env-file=/tmp/github_env_19922768520 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T09:06:00.3179117Z + container_name=13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T09:06:00.3179969Z + echo DOCKER_CONTAINER_ID=13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T09:06:00.3180946Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *\s\3\9\0\x* ]] 2025-12-04T09:06:00.3185376Z ++ echo dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl 2025-12-04T09:06:00.3188172Z + docker exec -t 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 sh -c 'python3 -m pip install dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-12-04T09:06:00.8072499Z Processing ./dist/torch-2.10.0a0+gitffd9b0f-cp310-cp310-linux_x86_64.whl (from torch==2.10.0a0+gitffd9b0f) 2025-12-04T09:06:01.9168626Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.18.0) 2025-12-04T09:06:01.9170292Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (4.12.2) 2025-12-04T09:06:01.9173290Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.13.3) 2025-12-04T09:06:01.9178366Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2.8.8) 2025-12-04T09:06:01.9182179Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.1.6) 2025-12-04T09:06:01.9187493Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (2025.10.0) 2025-12-04T09:06:01.9208017Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.3.0) 2025-12-04T09:06:01.9611336Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.22.4) 2025-12-04T09:06:01.9630084Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (1.3.0) 2025-12-04T09:06:01.9697890Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.10.0a0+gitffd9b0f->torch==2.10.0a0+gitffd9b0f) (3.0.3) 2025-12-04T09:06:02.4064650Z Installing collected packages: torch 2025-12-04T09:06:15.9086544Z Successfully installed torch-2.10.0a0+gitffd9b0f 2025-12-04T09:06:15.9704092Z + export TERM=vt100 2025-12-04T09:06:15.9704582Z + TERM=vt100 2025-12-04T09:06:15.9705010Z ++ dirname .ci/pytorch/test.sh 2025-12-04T09:06:15.9713645Z + source .ci/pytorch/common.sh 2025-12-04T09:06:15.9716524Z +++ dirname .ci/pytorch/common.sh 2025-12-04T09:06:15.9723915Z ++ source .ci/pytorch/common_utils.sh 2025-12-04T09:06:15.9725532Z +++ declare -f -t trap_add 2025-12-04T09:06:15.9735980Z ++ set -ex -o pipefail 2025-12-04T09:06:15.9736388Z ++ [[ linux-jammy-cuda12.8-py3.10-gcc11 == *rocm* ]] 2025-12-04T09:06:15.9736792Z ++ BUILD_TEST_LIBTORCH=0 2025-12-04T09:06:15.9737106Z ++ dirname .ci/pytorch/test.sh 2025-12-04T09:06:15.9743325Z + source .ci/pytorch/common-build.sh 2025-12-04T09:06:15.9744573Z ++ [[ linux-jammy-cuda12.8-py3.10-gcc11 != *win-* ]] 2025-12-04T09:06:15.9751408Z ++++ dirname .ci/pytorch/common-build.sh 2025-12-04T09:06:15.9759002Z +++ cd .ci/pytorch 2025-12-04T09:06:15.9759318Z +++ pwd -P 2025-12-04T09:06:15.9761587Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-12-04T09:06:15.9762113Z ++ [[ linux-jammy-cuda12.8-py3.10-gcc11 == *-pch* ]] 2025-12-04T09:06:15.9762494Z ++ which sccache 2025-12-04T09:06:15.9774148Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-12-04T09:06:15.9774590Z ++ sccache --stop-server 2025-12-04T09:06:15.9805669Z ++ true 2025-12-04T09:06:15.9806043Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-12-04T09:06:15.9810598Z ++ trap_add sccache_epilogue EXIT 2025-12-04T09:06:15.9810951Z ++ trap_add_cmd=sccache_epilogue 2025-12-04T09:06:15.9811255Z ++ shift 2025-12-04T09:06:15.9811503Z ++ for trap_add_name in "$@" 2025-12-04T09:06:15.9817233Z ++++ trap -p EXIT 2025-12-04T09:06:15.9819664Z +++ eval 'extract_trap_cmd ' 2025-12-04T09:06:15.9819976Z ++++ extract_trap_cmd 2025-12-04T09:06:15.9820685Z ++++ printf '%s\n' '' 2025-12-04T09:06:15.9821216Z +++ printf '%s\n' sccache_epilogue 2025-12-04T09:06:15.9821911Z ++ trap -- ' 2025-12-04T09:06:15.9822170Z sccache_epilogue' EXIT 2025-12-04T09:06:15.9822564Z ++ [[ -n 1 ]] 2025-12-04T09:06:15.9823007Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-12-04T09:06:15.9823877Z Skipping sccache server initialization, setting environment variables 2025-12-04T09:06:15.9824585Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:06:15.9824927Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:06:15.9825334Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:06:15.9825851Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:06:15.9830011Z ++ export RUST_LOG=sccache::server=error 2025-12-04T09:06:15.9830418Z ++ RUST_LOG=sccache::server=error 2025-12-04T09:06:15.9830774Z ++ sccache --zero-stats 2025-12-04T09:06:16.0915345Z Statistics zeroed. 2025-12-04T09:06:16.0921507Z ++ which ccache 2025-12-04T09:06:16.0932914Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 != *rocm* ]] 2025-12-04T09:06:16.0933459Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 != *s390x* ]] 2025-12-04T09:06:16.0933884Z + [[ -d /var/lib/jenkins/workspace ]] 2025-12-04T09:06:16.0935317Z ++ stat -c %u /var/lib/jenkins/workspace 2025-12-04T09:06:16.0950639Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-12-04T09:06:16.0951014Z + trap_add cleanup_workspace EXIT 2025-12-04T09:06:16.0951381Z + trap_add_cmd=cleanup_workspace 2025-12-04T09:06:16.0951695Z + shift 2025-12-04T09:06:16.0951959Z + for trap_add_name in "$@" 2025-12-04T09:06:16.0961210Z +++ trap -p EXIT 2025-12-04T09:06:16.0961578Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-12-04T09:06:16.0961957Z sccache_epilogue'\'' EXIT' 2025-12-04T09:06:16.0962464Z +++ extract_trap_cmd trap -- ' 2025-12-04T09:06:16.0962797Z sccache_epilogue' EXIT 2025-12-04T09:06:16.0963073Z +++ printf '%s\n' ' 2025-12-04T09:06:16.0963346Z sccache_epilogue' 2025-12-04T09:06:16.0963633Z ++ printf '%s\n' cleanup_workspace 2025-12-04T09:06:16.0963972Z + trap -- ' 2025-12-04T09:06:16.0964234Z sccache_epilogue 2025-12-04T09:06:16.0964517Z cleanup_workspace' EXIT 2025-12-04T09:06:16.0964865Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-12-04T09:06:16.7483649Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-12-04T09:06:16.7504201Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *cuda* ]] 2025-12-04T09:06:16.7505180Z ++ python -c 'import os;import numba.cuda; print(os.path.dirname(numba.cuda.__file__))' 2025-12-04T09:06:17.1874181Z + NUMBA_CUDA_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda 2025-12-04T09:06:17.1874941Z + '[' -n /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda ']' 2025-12-04T09:06:17.1879516Z +++ realpath .ci/pytorch/test.sh 2025-12-04T09:06:17.1888530Z ++ dirname /var/lib/jenkins/workspace/.ci/pytorch/test.sh 2025-12-04T09:06:17.1896143Z + NUMBA_PATCH=/var/lib/jenkins/workspace/.ci/pytorch/numba-cuda-13.patch 2025-12-04T09:06:17.1896792Z + pushd /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda 2025-12-04T09:06:17.1897825Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda ~/workspace 2025-12-04T09:06:17.1898359Z + patch -p4 2025-12-04T09:06:17.1909999Z patching file cudadrv/driver.py 2025-12-04T09:06:17.1910383Z Hunk #1 succeeded at 357 (offset -8 lines). 2025-12-04T09:06:17.1928183Z + popd 2025-12-04T09:06:17.1928425Z ~/workspace 2025-12-04T09:06:17.1928692Z + echo 'Environment variables:' 2025-12-04T09:06:17.1929033Z Environment variables: 2025-12-04T09:06:17.1929312Z + env 2025-12-04T09:06:17.1933551Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:06:17.1934106Z CONTINUE_THROUGH_ERROR=True 2025-12-04T09:06:17.1934494Z BUILD_ENVIRONMENT=linux-jammy-cuda12.8-py3.10-gcc11 2025-12-04T09:06:17.1935169Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T09:06:17.1935504Z HOSTNAME=13d84695993c 2025-12-04T09:06:17.1936171Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.1937191Z GITHUB_ACTION=__run_3 2025-12-04T09:06:17.1937499Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 2025-12-04T09:06:17.1937824Z GITHUB_RUN_NUMBER=158165 2025-12-04T09:06:17.1938116Z TEST_CONFIG=distributed 2025-12-04T09:06:17.1938419Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T09:06:17.1938766Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T09:06:17.1939118Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:06:17.1939556Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T09:06:17.1939889Z GITHUB_TRIGGERING_ACTOR=huydhn 2025-12-04T09:06:17.1940192Z GITHUB_REF_TYPE=branch 2025-12-04T09:06:17.1940526Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:06:17.1940907Z XLA_CUDA= 2025-12-04T09:06:17.1941150Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T09:06:17.1941599Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T09:06:17.1942071Z *** 2025-12-04T09:06:17.1942316Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T09:06:17.1942617Z GITHUB_ACTIONS=true 2025-12-04T09:06:17.1942900Z NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:06:17.1943291Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:06:17.1943734Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:06:17.1944164Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:06:17.1944746Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/heads/main 2025-12-04T09:06:17.1945267Z UCC_HOME=/usr 2025-12-04T09:06:17.1945522Z VERBOSE_TEST_LOGS=False 2025-12-04T09:06:17.1945813Z GITHUB_REF=refs/heads/main 2025-12-04T09:06:17.1946093Z SHARD_NUMBER=2 2025-12-04T09:06:17.1946352Z GITHUB_REF_PROTECTED=true 2025-12-04T09:06:17.1946645Z HOME=/var/lib/jenkins 2025-12-04T09:06:17.1946943Z GITHUB_API_URL=https://api.github.com 2025-12-04T09:06:17.1947397Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T09:06:17.1947973Z UCX_COMMIT=7836b165abdbe468a2f607e7254011c07d788152 2025-12-04T09:06:17.1948368Z USE_SYSTEM_NCCL=1 2025-12-04T09:06:17.1948620Z NUM_TEST_SHARDS=3 2025-12-04T09:06:17.1948898Z UCX_HOME=/usr 2025-12-04T09:06:17.1949569Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.1950686Z JOB_NAME=linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T09:06:17.1951769Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.1952715Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T09:06:17.1953308Z GITHUB_EVENT_NAME=schedule 2025-12-04T09:06:17.1953599Z DASHBOARD_TAG= 2025-12-04T09:06:17.1953867Z GITHUB_RUN_ID=19922768520 2025-12-04T09:06:17.1954164Z INSTALLED_OPENBLAS= 2025-12-04T09:06:17.1954876Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.1955669Z GITHUB_ACTOR=huydhn 2025-12-04T09:06:17.1955934Z PR_NUMBER= 2025-12-04T09:06:17.1956176Z DESIRED_CUDA=12.8.1 2025-12-04T09:06:17.1956444Z GITHUB_RUN_ATTEMPT=1 2025-12-04T09:06:17.1956872Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T09:06:17.1957264Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T09:06:17.1957653Z TERM=vt100 2025-12-04T09:06:17.1957897Z INSTALLED_VISION=yes 2025-12-04T09:06:17.1958171Z BRANCH=main 2025-12-04T09:06:17.1958411Z SCCACHE_REGION=us-east-1 2025-12-04T09:06:17.1958726Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T09:06:17.1959168Z BUILD_AOT_INDUCTOR_TEST= 2025-12-04T09:06:17.1959452Z CUDA_PATH=/usr/local/cuda 2025-12-04T09:06:17.1960039Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T09:06:17.1960699Z GITHUB_SERVER_URL=https://github.com 2025-12-04T09:06:17.1961084Z UCC_COMMIT=430e241bf5d38cbc73fc7a6b89155397232e3f96 2025-12-04T09:06:17.1961472Z REENABLED_ISSUES= 2025-12-04T09:06:17.1961721Z DOCS= 2025-12-04T09:06:17.1961928Z SHLVL=1 2025-12-04T09:06:17.1962146Z MAX_JOBS=46 2025-12-04T09:06:17.1962386Z GITHUB_ACTOR_ID=475357 2025-12-04T09:06:17.1962860Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:06:17.1963290Z GITHUB_REF_NAME=main 2025-12-04T09:06:17.1963718Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:06:17.1964203Z GITHUB_JOB=test 2025-12-04T09:06:17.1964450Z NO_TEST_TIMEOUT=False 2025-12-04T09:06:17.1964730Z TD_DISTRIBUTED=False 2025-12-04T09:06:17.1965028Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T09:06:17.1965354Z GITHUB_RETENTION_DAYS=90 2025-12-04T09:06:17.1965650Z OPENSSL_DIR=/opt/openssl 2025-12-04T09:06:17.1965951Z GITHUB_ACTION_REPOSITORY= 2025-12-04T09:06:17.1966824Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:06:17.1967745Z GITHUB_BASE_REF= 2025-12-04T09:06:17.1968004Z INSTALLED_ACL= 2025-12-04T09:06:17.1968518Z ARTIFACTS_FILE_SUFFIX=test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912 2025-12-04T09:06:17.1969098Z CI=true 2025-12-04T09:06:17.1969351Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T09:06:17.1969713Z RUST_LOG=sccache::server=error 2025-12-04T09:06:17.1970004Z JOB_ID=57116084912 2025-12-04T09:06:17.1970262Z GITHUB_HEAD_REF= 2025-12-04T09:06:17.1970520Z GITHUB_ACTION_REF= 2025-12-04T09:06:17.1970830Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T09:06:17.1971224Z TEST_SHOWLOCALS=False 2025-12-04T09:06:17.1971503Z GITHUB_WORKFLOW=trunk 2025-12-04T09:06:17.1971784Z DEBIAN_FRONTEND=noninteractive 2025-12-04T09:06:17.1972498Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.1973225Z NO_TD=False 2025-12-04T09:06:17.1973473Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T09:06:17.1973817Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T09:06:17.1974165Z _=/usr/bin/env 2025-12-04T09:06:17.1974562Z OLDPWD=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda 2025-12-04T09:06:17.1975145Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-12-04T09:06:17.2071300Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-12-04T09:06:17.2072036Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-12-04T09:06:17.2072709Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-12-04T09:06:17.2073408Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-12-04T09:06:17.2073935Z + BUILD_DIR=build 2025-12-04T09:06:17.2074222Z + BUILD_RENAMED_DIR=build_renamed 2025-12-04T09:06:17.2074566Z + BUILD_BIN_DIR=build/bin 2025-12-04T09:06:17.2074863Z + SHARD_NUMBER=2 2025-12-04T09:06:17.2075129Z + NUM_TEST_SHARDS=3 2025-12-04T09:06:17.2075413Z + export TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:06:17.2075776Z + TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:06:17.2076102Z + export VALGRIND=ON 2025-12-04T09:06:17.2076367Z + VALGRIND=ON 2025-12-04T09:06:17.2076706Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *clang9* ]] 2025-12-04T09:06:17.2077369Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *xpu* ]] 2025-12-04T09:06:17.2077766Z + detect_cuda_arch 2025-12-04T09:06:17.2078096Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *cuda* ]] 2025-12-04T09:06:17.2078506Z + command -v nvidia-smi 2025-12-04T09:06:17.2078802Z /usr/bin/nvidia-smi 2025-12-04T09:06:17.2080550Z ++ nvidia-smi --query-gpu=compute_cap --format=csv 2025-12-04T09:06:17.2081386Z ++ tail -n 1 2025-12-04T09:06:17.2579646Z + TORCH_CUDA_ARCH_LIST=7.5 2025-12-04T09:06:17.2580064Z + export TORCH_CUDA_ARCH_LIST 2025-12-04T09:06:17.2580450Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *s390x* ]] 2025-12-04T09:06:17.2580856Z + [[ 0 == \1 ]] 2025-12-04T09:06:17.2581106Z + [[ True == \1 ]] 2025-12-04T09:06:17.2581411Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 != *bazel* ]] 2025-12-04T09:06:17.2582184Z ++ realpath build/custom_test_artifacts 2025-12-04T09:06:17.2589771Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-12-04T09:06:17.2590633Z + [[ -n '' ]] 2025-12-04T09:06:17.2590927Z + echo 'Environment variables' 2025-12-04T09:06:17.2591255Z Environment variables 2025-12-04T09:06:17.2591518Z + env 2025-12-04T09:06:17.2597374Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T09:06:17.2597891Z CONTINUE_THROUGH_ERROR=True 2025-12-04T09:06:17.2598264Z BUILD_ENVIRONMENT=linux-jammy-cuda12.8-py3.10-gcc11 2025-12-04T09:06:17.2598937Z VLLM_TEST_HUGGING_FACE_TOKEN=*** 2025-12-04T09:06:17.2599298Z HOSTNAME=13d84695993c 2025-12-04T09:06:17.2599978Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.2600815Z GITHUB_ACTION=__run_3 2025-12-04T09:06:17.2601111Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 2025-12-04T09:06:17.2601457Z GITHUB_RUN_NUMBER=158165 2025-12-04T09:06:17.2601754Z TEST_CONFIG=distributed 2025-12-04T09:06:17.2602045Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-12-04T09:06:17.2602420Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-12-04T09:06:17.2602783Z SCCACHE_IDLE_TIMEOUT=0 2025-12-04T09:06:17.2603224Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-12-04T09:06:17.2603557Z GITHUB_TRIGGERING_ACTOR=huydhn 2025-12-04T09:06:17.2603867Z GITHUB_REF_TYPE=branch 2025-12-04T09:06:17.2604139Z TORCH_CUDA_ARCH_LIST=7.5 2025-12-04T09:06:17.2604483Z BASE_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:06:17.2604864Z XLA_CUDA= 2025-12-04T09:06:17.2605107Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-12-04T09:06:17.2605618Z HUGGING_FACE_HUB_TOKEN=*** 2025-12-04T09:06:17.2605972Z *** 2025-12-04T09:06:17.2606216Z GITHUB_REPOSITORY_ID=65600975 2025-12-04T09:06:17.2606516Z GITHUB_ACTIONS=true 2025-12-04T09:06:17.2606798Z NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T09:06:17.2607178Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-12-04T09:06:17.2607641Z SHA1=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:06:17.2608061Z GITHUB_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:06:17.2608659Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/heads/main 2025-12-04T09:06:17.2609197Z UCC_HOME=/usr 2025-12-04T09:06:17.2609449Z TORCH_SERIALIZATION_DEBUG=1 2025-12-04T09:06:17.2609761Z VERBOSE_TEST_LOGS=False 2025-12-04T09:06:17.2610055Z GITHUB_REF=refs/heads/main 2025-12-04T09:06:17.2610337Z SHARD_NUMBER=2 2025-12-04T09:06:17.2610603Z GITHUB_REF_PROTECTED=true 2025-12-04T09:06:17.2610904Z HOME=/var/lib/jenkins 2025-12-04T09:06:17.2611204Z GITHUB_API_URL=https://api.github.com 2025-12-04T09:06:17.2611580Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-12-04T09:06:17.2611974Z UCX_COMMIT=7836b165abdbe468a2f607e7254011c07d788152 2025-12-04T09:06:17.2612350Z USE_SYSTEM_NCCL=1 2025-12-04T09:06:17.2612613Z NUM_TEST_SHARDS=3 2025-12-04T09:06:17.2612869Z UCX_HOME=/usr 2025-12-04T09:06:17.2613508Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.2614585Z JOB_NAME=linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T09:06:17.2615793Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.2616720Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-12-04T09:06:17.2617293Z GITHUB_EVENT_NAME=schedule 2025-12-04T09:06:17.2617576Z DASHBOARD_TAG= 2025-12-04T09:06:17.2617832Z GITHUB_RUN_ID=19922768520 2025-12-04T09:06:17.2618123Z INSTALLED_OPENBLAS= 2025-12-04T09:06:17.2618808Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.2619578Z GITHUB_ACTOR=huydhn 2025-12-04T09:06:17.2619834Z PR_NUMBER= 2025-12-04T09:06:17.2620058Z DESIRED_CUDA=12.8.1 2025-12-04T09:06:17.2620322Z GITHUB_RUN_ATTEMPT=1 2025-12-04T09:06:17.2620587Z VALGRIND=ON 2025-12-04T09:06:17.2620827Z ANACONDA_PYTHON_VERSION=3.10 2025-12-04T09:06:17.2621204Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-12-04T09:06:17.2621694Z TERM=vt100 2025-12-04T09:06:17.2621922Z INSTALLED_VISION=yes 2025-12-04T09:06:17.2622189Z BRANCH=main 2025-12-04T09:06:17.2622439Z SCCACHE_REGION=us-east-1 2025-12-04T09:06:17.2622731Z OPENSSL_ROOT_DIR=/opt/openssl 2025-12-04T09:06:17.2623051Z BUILD_AOT_INDUCTOR_TEST= 2025-12-04T09:06:17.2623378Z CUDA_PATH=/usr/local/cuda 2025-12-04T09:06:17.2624155Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-12-04T09:06:17.2625009Z GITHUB_SERVER_URL=https://github.com 2025-12-04T09:06:17.2625422Z UCC_COMMIT=430e241bf5d38cbc73fc7a6b89155397232e3f96 2025-12-04T09:06:17.2625820Z REENABLED_ISSUES= 2025-12-04T09:06:17.2626063Z DOCS= 2025-12-04T09:06:17.2626287Z SHLVL=1 2025-12-04T09:06:17.2626512Z MAX_JOBS=46 2025-12-04T09:06:17.2626746Z GITHUB_ACTOR_ID=475357 2025-12-04T09:06:17.2627139Z GITHUB_WORKFLOW_SHA=ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T09:06:17.2627688Z GITHUB_REF_NAME=main 2025-12-04T09:06:17.2628142Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-12-04T09:06:17.2628632Z GITHUB_JOB=test 2025-12-04T09:06:17.2628903Z NO_TEST_TIMEOUT=False 2025-12-04T09:06:17.2629195Z TD_DISTRIBUTED=False 2025-12-04T09:06:17.2629490Z GITHUB_REPOSITORY=pytorch/pytorch 2025-12-04T09:06:17.2629837Z GITHUB_RETENTION_DAYS=90 2025-12-04T09:06:17.2630141Z OPENSSL_DIR=/opt/openssl 2025-12-04T09:06:17.2630433Z GITHUB_ACTION_REPOSITORY= 2025-12-04T09:06:17.2631359Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:06:17.2632313Z GITHUB_BASE_REF= 2025-12-04T09:06:17.2632562Z INSTALLED_ACL= 2025-12-04T09:06:17.2633088Z ARTIFACTS_FILE_SUFFIX=test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912 2025-12-04T09:06:17.2633689Z CI=true 2025-12-04T09:06:17.2633946Z GITHUB_REPOSITORY_OWNER=pytorch 2025-12-04T09:06:17.2634301Z RUST_LOG=sccache::server=error 2025-12-04T09:06:17.2634618Z JOB_ID=57116084912 2025-12-04T09:06:17.2634885Z GITHUB_HEAD_REF= 2025-12-04T09:06:17.2635136Z GITHUB_ACTION_REF= 2025-12-04T09:06:17.2635471Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-12-04T09:06:17.2635885Z TEST_SHOWLOCALS=False 2025-12-04T09:06:17.2636160Z GITHUB_WORKFLOW=trunk 2025-12-04T09:06:17.2636459Z DEBIAN_FRONTEND=noninteractive 2025-12-04T09:06:17.2637192Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a69b4c3d-7d46-4b0a-9f5f-55b0c4714070 2025-12-04T09:06:17.2637927Z NO_TD=False 2025-12-04T09:06:17.2638193Z SKIP_SCCACHE_INITIALIZATION=1 2025-12-04T09:06:17.2638551Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-12-04T09:06:17.2639172Z OLDPWD=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/numba/cuda 2025-12-04T09:06:17.2639653Z _=/usr/bin/env 2025-12-04T09:06:17.2639906Z + echo 'Testing pytorch' 2025-12-04T09:06:17.2640195Z Testing pytorch 2025-12-04T09:06:17.2640448Z + export LANG=C.UTF-8 2025-12-04T09:06:17.2640732Z + LANG=C.UTF-8 2025-12-04T09:06:17.2640979Z + PR_NUMBER= 2025-12-04T09:06:17.2641364Z + [[ distributed == \d\e\f\a\u\l\t ]] 2025-12-04T09:06:17.2641729Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-12-04T09:06:17.2642143Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *rocm* ]] 2025-12-04T09:06:17.2642529Z + [[ distributed == \s\l\o\w ]] 2025-12-04T09:06:17.2642940Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *slow-gradcheck* ]] 2025-12-04T09:06:17.2643425Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *cuda* ]] 2025-12-04T09:06:17.2643835Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-12-04T09:06:17.2644225Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-12-04T09:06:17.2644583Z + [[ distributed == *crossref* ]] 2025-12-04T09:06:17.2644957Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *rocm* ]] 2025-12-04T09:06:17.2645380Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *xpu* ]] 2025-12-04T09:06:17.2645824Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 != *-bazel-* ]] 2025-12-04T09:06:17.2646322Z + pip_install ninja==1.10.2 2025-12-04T09:06:17.2647072Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-12-04T09:06:17.2647708Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-12-04T09:06:17.6506606Z Collecting ninja==1.10.2 2025-12-04T09:06:17.6750396Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-12-04T09:06:17.6851354Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-12-04T09:06:18.0826204Z Installing collected packages: ninja 2025-12-04T09:06:18.0826916Z Attempting uninstall: ninja 2025-12-04T09:06:18.0831656Z Found existing installation: ninja 1.11.1.4 2025-12-04T09:06:18.0858557Z Uninstalling ninja-1.11.1.4: 2025-12-04T09:06:18.0937443Z Successfully uninstalled ninja-1.11.1.4 2025-12-04T09:06:18.1252203Z Successfully installed ninja-1.10.2 2025-12-04T09:06:18.1791538Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:06:18.1793483Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-12-04T09:06:18.1794681Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *aarch64* ]] 2025-12-04T09:06:18.1795164Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *asan* ]] 2025-12-04T09:06:18.1795635Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *-debug* ]] 2025-12-04T09:06:18.1796094Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 != *-bazel-* ]] 2025-12-04T09:06:18.1796754Z + echo 'We are not in debug mode: linux-jammy-cuda12.8-py3.10-gcc11. Expect the assertion to pass' 2025-12-04T09:06:18.1797571Z We are not in debug mode: linux-jammy-cuda12.8-py3.10-gcc11. Expect the assertion to pass 2025-12-04T09:06:18.1798125Z + cd test 2025-12-04T09:06:18.1798528Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-12-04T09:06:19.8826606Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-12-04T09:06:19.8827121Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-12-04T09:06:19.8827714Z + [[ distributed == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-12-04T09:06:19.8830946Z + DYNAMO_BENCHMARK_FLAGS=() 2025-12-04T09:06:19.8831688Z + [[ distributed == *pr_time_benchmarks* ]] 2025-12-04T09:06:19.8832085Z + [[ distributed == *dynamo_eager* ]] 2025-12-04T09:06:19.8832468Z + [[ distributed == *aot_eager* ]] 2025-12-04T09:06:19.8832810Z + [[ distributed == *aot_inductor* ]] 2025-12-04T09:06:19.8833194Z + [[ distributed == *max_autotune_inductor* ]] 2025-12-04T09:06:19.8833589Z + [[ distributed == *inductor* ]] 2025-12-04T09:06:19.8833920Z + [[ distributed == *dynamic* ]] 2025-12-04T09:06:19.8834260Z + [[ distributed == *cpu* ]] 2025-12-04T09:06:19.8834580Z + [[ distributed == *xpu* ]] 2025-12-04T09:06:19.8834944Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2025-12-04T09:06:19.8863803Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *libtorch* ]] 2025-12-04T09:06:19.8864792Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *-bazel-* ]] 2025-12-04T09:06:19.8865191Z + cd test 2025-12-04T09:06:19.8865530Z + python -c 'import torch; print(torch.__config__.show())' 2025-12-04T09:06:22.1279194Z PyTorch built with: 2025-12-04T09:06:22.1279558Z - GCC 11.4 2025-12-04T09:06:22.1279851Z - C++ Version: 201703 2025-12-04T09:06:22.1281045Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T09:06:22.1281875Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T09:06:22.1282378Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T09:06:22.1282750Z - LAPACK is enabled (usually provided by MKL) 2025-12-04T09:06:22.1283130Z - NNPACK is enabled 2025-12-04T09:06:22.1283432Z - CPU capability usage: AVX512 2025-12-04T09:06:22.1283742Z - CUDA Runtime 12.8 2025-12-04T09:06:22.1284304Z - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_89,code=sm_89 2025-12-04T09:06:22.1285257Z - CuDNN 91.0.2 (built against CUDA 12.9) 2025-12-04T09:06:22.1290988Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=35b7a9a26c5923d98aebaa41a031dae21788a9ee, CUDA_VERSION=12.8, CUDNN_VERSION=9.10.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-12-04T09:06:22.1296430Z 2025-12-04T09:06:22.6020210Z + cd test 2025-12-04T09:06:22.6020786Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-12-04T09:06:24.0262279Z ATen/Parallel: 2025-12-04T09:06:24.0262792Z at::get_num_threads() : 24 2025-12-04T09:06:24.0263135Z at::get_num_interop_threads() : 24 2025-12-04T09:06:24.0263502Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-12-04T09:06:24.0263836Z omp_get_max_threads() : 24 2025-12-04T09:06:24.0264479Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-12-04T09:06:24.0265151Z mkl_get_max_threads() : 24 2025-12-04T09:06:24.0265629Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-12-04T09:06:24.0266138Z std::thread::hardware_concurrency() : 48 2025-12-04T09:06:24.0266489Z Environment variables: 2025-12-04T09:06:24.0266790Z OMP_NUM_THREADS : [not set] 2025-12-04T09:06:24.0267105Z MKL_NUM_THREADS : [not set] 2025-12-04T09:06:24.0267516Z ATen parallel backend: OpenMP 2025-12-04T09:06:24.0267913Z 2025-12-04T09:06:24.3017699Z + [[ distributed == *numpy_2* ]] 2025-12-04T09:06:24.3018297Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *aarch64* ]] 2025-12-04T09:06:24.3018712Z + [[ distributed == *backward* ]] 2025-12-04T09:06:24.3019108Z + [[ distributed == *libtorch_agnostic_targetting* ]] 2025-12-04T09:06:24.3019516Z + [[ distributed == *xla* ]] 2025-12-04T09:06:24.3019818Z + [[ distributed == *vllm* ]] 2025-12-04T09:06:24.3020145Z + [[ distributed == *executorch* ]] 2025-12-04T09:06:24.3020501Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2025-12-04T09:06:24.3020882Z + [[ distributed == \q\u\a\n\t\i\z\a\t\i\o\n ]] 2025-12-04T09:06:24.3022977Z + [[ linux-jammy-cuda12.8-py3.10-gcc11 == *libtorch* ]] 2025-12-04T09:06:24.3023509Z + [[ distributed == distributed ]] 2025-12-04T09:06:24.3024069Z + test_distributed 2025-12-04T09:06:24.3024353Z + echo 'Testing distributed python tests' 2025-12-04T09:06:24.3024908Z Testing distributed python tests 2025-12-04T09:06:24.3025394Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2025-12-04T09:06:29.8594285Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2025-12-04T09:06:29.9116855Z Ignoring disabled issues: [''] 2025-12-04T09:06:29.9223982Z Found test times from artifacts 2025-12-04T09:06:29.9638120Z Found test times from artifacts 2025-12-04T09:06:29.9656500Z Running all tests 2025-12-04T09:06:29.9826452Z Running parallel tests on 1 processes 2025-12-04T09:06:29.9831044Z Name: tests to run (est. time: 135.15min) 2025-12-04T09:06:29.9831709Z Serial tests (93): 2025-12-04T09:06:29.9832044Z distributed/test_dynamo_distributed 2/2 2025-12-04T09:06:29.9832436Z distributed/fsdp/test_fsdp_uneven 1/1 2025-12-04T09:06:29.9832827Z distributed/fsdp/test_fsdp_input 1/1 2025-12-04T09:06:29.9833223Z distributed/fsdp/test_fsdp_traversal 1/1 2025-12-04T09:06:29.9833624Z distributed/fsdp/test_fsdp_checkpoint 1/1 2025-12-04T09:06:29.9834023Z distributed/fsdp/test_fsdp_comm 1/1 2025-12-04T09:06:29.9834495Z distributed/fsdp/test_distributed_checkpoint 1/1 2025-12-04T09:06:29.9834930Z distributed/test_c10d_ops_nccl 1/1 2025-12-04T09:06:29.9835346Z distributed/elastic/multiprocessing/api_test 1/1 2025-12-04T09:06:29.9835801Z distributed/test_inductor_collectives 1/1 2025-12-04T09:06:29.9836228Z distributed/tensor/test_dtensor_export 1/1 2025-12-04T09:06:29.9836657Z distributed/_tools/test_fake_collectives 1/1 2025-12-04T09:06:29.9837068Z distributed/test_control_collectives 1/1 2025-12-04T09:06:29.9837466Z distributed/test_collective_utils 1/1 2025-12-04T09:06:29.9837960Z distributed/algorithms/quantization/test_quantization 1/1 2025-12-04T09:06:29.9838457Z distributed/test_c10d_object_collectives 1/1 2025-12-04T09:06:29.9838898Z distributed/tensor/debug/test_op_coverage 1/1 2025-12-04T09:06:29.9839499Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2025-12-04T09:06:29.9839941Z distributed/_tools/test_mod_tracker 1/1 2025-12-04T09:06:29.9840365Z distributed/_shard/sharded_tensor/test_logger 1/1 2025-12-04T09:06:29.9840800Z distributed/tensor/test_dtensor_compile 1/1 2025-12-04T09:06:29.9841227Z distributed/test_aten_comm_compute_reordering 2/2 2025-12-04T09:06:29.9841662Z distributed/_tools/test_sac_estimator 1/1 2025-12-04T09:06:29.9842060Z distributed/_tools/test_memory_tracker 1/1 2025-12-04T09:06:29.9842519Z distributed/checkpoint/_experimental/test_builder 1/1 2025-12-04T09:06:29.9843003Z distributed/_composable/test_replicate_with_fsdp 1/1 2025-12-04T09:06:29.9843464Z distributed/tensor/test_xla_integration 1/1 2025-12-04T09:06:29.9843916Z distributed/checkpoint/_experimental/test_types 1/1 2025-12-04T09:06:29.9844420Z distributed/tensor/experimental/test_register_sharding 1/1 2025-12-04T09:06:29.9844881Z distributed/test_backends 1/1 2025-12-04T09:06:29.9845255Z distributed/tensor/test_experimental_ops 1/1 2025-12-04T09:06:29.9845713Z distributed/checkpoint/test_quantized_hf_storage 1/1 2025-12-04T09:06:29.9846269Z distributed/_composable/test_composability/test_pp_composability 1/1 2025-12-04T09:06:29.9846857Z distributed/checkpoint/test_async_process_executor 1/1 2025-12-04T09:06:29.9847304Z distributed/tensor/test_tensor_ops 1/1 2025-12-04T09:06:29.9847665Z distributed/test_device_mesh 1/1 2025-12-04T09:06:29.9848044Z distributed/optim/test_named_optimizer 1/1 2025-12-04T09:06:29.9848541Z distributed/_composable/fsdp/test_fully_shard_ignore_params 1/1 2025-12-04T09:06:29.9849078Z distributed/checkpoint/_experimental/test_staging 1/1 2025-12-04T09:06:29.9849766Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2025-12-04T09:06:29.9850273Z distributed/tensor/test_embedding_ops 1/1 2025-12-04T09:06:29.9850711Z distributed/tensor/experimental/test_local_map 1/1 2025-12-04T09:06:29.9851121Z distributed/test_local_tensor 1/1 2025-12-04T09:06:29.9851554Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2025-12-04T09:06:29.9852033Z distributed/checkpoint/test_tp_checkpoint 1/1 2025-12-04T09:06:29.9852436Z distributed/pipelining/test_stage 1/1 2025-12-04T09:06:29.9852870Z distributed/tensor/parallel/test_tp_random_state 1/1 2025-12-04T09:06:29.9853311Z distributed/checkpoint/test_planner 1/1 2025-12-04T09:06:29.9853727Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2025-12-04T09:06:29.9854162Z distributed/pipelining/test_schedule 1/1 2025-12-04T09:06:29.9854620Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2025-12-04T09:06:29.9855132Z distributed/test_run 1/1 2025-12-04T09:06:29.9855455Z distributed/tensor/test_math_ops 1/1 2025-12-04T09:06:29.9855822Z distributed/fsdp/test_utils 1/1 2025-12-04T09:06:29.9856328Z distributed/tensor/parallel/test_tp_examples 1/1 2025-12-04T09:06:29.9856812Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 2025-12-04T09:06:29.9857414Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2025-12-04T09:06:29.9857951Z distributed/checkpoint/test_pg_transport 1/1 2025-12-04T09:06:29.9858477Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2025-12-04T09:06:29.9859017Z distributed/checkpoint/test_utils 1/1 2025-12-04T09:06:29.9859586Z distributed/checkpoint/_experimental/test_checkpoint_process 1/1 2025-12-04T09:06:29.9860153Z distributed/tensor/test_view_ops 1/1 2025-12-04T09:06:29.9860525Z distributed/fsdp/test_fsdp_state_dict 2/2 2025-12-04T09:06:29.9860980Z distributed/test_distributed_spawn 2/9 2025-12-04T09:06:29.9861371Z distributed/test_distributed_spawn 5/9 2025-12-04T09:06:29.9861815Z distributed/test_distributed_spawn 8/9 2025-12-04T09:06:29.9862217Z distributed/fsdp/test_checkpoint_wrapper 1/1 2025-12-04T09:06:29.9862756Z distributed/checkpoint/test_file_system_checkpoint 1/1 2025-12-04T09:06:29.9863271Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2025-12-04T09:06:29.9863754Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2025-12-04T09:06:29.9864140Z distributed/_shard/test_sharder 1/1 2025-12-04T09:06:29.9864589Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2025-12-04T09:06:29.9865129Z distributed/fsdp/test_fsdp_tp_integration 1/1 2025-12-04T09:06:29.9865592Z distributed/test_c10d_pypg 1/1 2025-12-04T09:06:29.9865944Z distributed/test_pg_wrapper 1/1 2025-12-04T09:06:29.9866388Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2025-12-04T09:06:29.9866876Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2025-12-04T09:06:29.9867507Z distributed/nn/jit/test_instantiator 1/1 2025-12-04T09:06:29.9868211Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2025-12-04T09:06:29.9868723Z distributed/test_nccl 1/1 2025-12-04T09:06:29.9869064Z distributed/fsdp/test_fsdp_misc 1/1 2025-12-04T09:06:29.9869492Z distributed/fsdp/test_fsdp_meta 1/1 2025-12-04T09:06:29.9869873Z distributed/test_data_parallel 1/1 2025-12-04T09:06:29.9870348Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 2025-12-04T09:06:29.9870803Z distributed/fsdp/test_fsdp_unshard_params 1/1 2025-12-04T09:06:29.9871328Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2025-12-04T09:06:29.9871922Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2025-12-04T09:06:29.9872413Z distributed/test_c10d_spawn_gloo 1/1 2025-12-04T09:06:29.9872771Z distributed/test_c10d_ucc 1/1 2025-12-04T09:06:29.9873159Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2025-12-04T09:06:29.9873573Z distributed/test_c10d_common 1/1 2025-12-04T09:06:29.9873966Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2025-12-04T09:06:29.9874484Z distributed/test_c10d_nccl 2/3 2025-12-04T09:06:29.9874826Z Parallel tests (0): 2025-12-04T09:06:29.9875116Z Name: excluded (est. time: 0.0min) 2025-12-04T09:06:29.9875451Z Serial tests (0): 2025-12-04T09:06:29.9875723Z Parallel tests (0): 2025-12-04T09:06:29.9876267Z Running distributed/test_dynamo_distributed 2/2 ... [2025-12-04 09:06:29.983902][820.921541282] 2025-12-04T09:06:29.9876867Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:06:29.9878181Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:06:29.984258] 2025-12-04T09:11:16.6093426Z 2025-12-04T09:11:16.6094542Z distributed/test_dynamo_distributed 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_2.2_9c1eb87275ed5b08_.log 2025-12-04T09:11:16.6107838Z Running 24 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_multiproc_autotune_dynamic_shapes, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training 2025-12-04T09:11:16.6119921Z 2025-12-04T09:11:16.6120337Z Finished distributed/test_dynamo_distributed 2/2 ... [2025-12-04 09:11:16.608636][1107.546268343], took 4.78min 2025-12-04T09:11:16.6121746Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-98b5e89a175f7fbc.xml 2025-12-04T09:11:17.0668607Z Uploading artifacts took 0.16 seconds 2025-12-04T09:11:17.0670193Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2025-12-04 09:11:17.066864][1108.004495732] 2025-12-04T09:11:17.0670898Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:11:17.0674787Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:11:17.067214] 2025-12-04T09:12:01.0338391Z 2025-12-04T09:12:01.0339309Z PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven 1/1 (test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_04d990d4b348b53e_.log) 2025-12-04T09:12:01.0340694Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-bb030510459950ee.xml 2025-12-04T09:12:01.0341627Z ============================= test session starts ============================== 2025-12-04T09:12:01.0342296Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:12:01.0342892Z cachedir: .pytest_cache 2025-12-04T09:12:01.0343953Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:12:01.0344725Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:12:01.0345084Z configfile: pytest.ini 2025-12-04T09:12:01.0345780Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:12:01.0346554Z collecting ... collected 1 item 2025-12-04T09:12:01.0346949Z stepcurrent: Cannot find last run test, not skipping 2025-12-04T09:12:01.0348120Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda 2025-12-04T09:12:01.0348769Z 2025-12-04T09:12:01.0349739Z distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda I1204 09:11:20.484000 20927 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 20979 2025-12-04T09:12:01.0351346Z I1204 09:11:20.485000 20927 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 20980 2025-12-04T09:12:01.0352479Z I1204 09:11:20.486000 20927 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 20981 2025-12-04T09:12:01.0353615Z I1204 09:11:20.486000 20927 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 20982 2025-12-04T09:12:01.0354701Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0355838Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0357524Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0359208Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0360931Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0362419Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0364047Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0365603Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0367314Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0368868Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0370405Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0371909Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0373427Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0375056Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0377200Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 2. CUDA driver allocated memory was 604962816 and is now 628031488. 2025-12-04T09:12:01.0379197Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0380337Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0382105Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0383585Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0384773Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0386119Z [rank2]:E1204 09:11:27.262000 20981 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:12:01.0387319Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0388626Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0390309Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0391959Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0393614Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0395181Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0396679Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0398368Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0400063Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0401611Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0403148Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0404666Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0406193Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0407811Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0409959Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 1. CUDA driver allocated memory was 602865664 and is now 628031488. 2025-12-04T09:12:01.0411960Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0413088Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0414868Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0416349Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0417546Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0418903Z [rank1]:E1204 09:11:27.262000 20980 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:12:01.0420017Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0421113Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0422747Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0424787Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0426427Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0428054Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0429716Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0431324Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0432906Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0434502Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0436101Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0437649Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0439394Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0441034Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0443103Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 3. CUDA driver allocated memory was 489619456 and is now 628031488. 2025-12-04T09:12:01.0445050Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0446166Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0447875Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0449296Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0450457Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0451778Z [rank3]:E1204 09:11:27.263000 20982 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:12:01.0452855Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0453925Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0455491Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0457051Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0458596Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0460041Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0461680Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0463180Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0464684Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0466190Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0467943Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0469559Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0471120Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0472726Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0474917Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 714014720 and is now 737083392. 2025-12-04T09:12:01.0476989Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0478142Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0480035Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0481468Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0482626Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0483947Z [rank0]:E1204 09:11:27.264000 20979 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:12:01.0484688Z dist init r=2, world=4 2025-12-04T09:12:01.0484960Z dist init r=1, world=4 2025-12-04T09:12:01.0485227Z dist init r=3, world=4 2025-12-04T09:12:01.0485479Z dist init r=0, world=4 2025-12-04T09:12:01.0486741Z [rank0]:[W1204 09:11:27.616896860 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:12:01.0488047Z FAILED [8.3006s] [100%] 2025-12-04T09:12:01.0488213Z 2025-12-04T09:12:01.0488370Z =================================== FAILURES =================================== 2025-12-04T09:12:01.0488909Z _______________ TestUnevenParamShardCUDA.test_one_iteration_cuda _______________ 2025-12-04T09:12:01.0489423Z Traceback (most recent call last): 2025-12-04T09:12:01.0490228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:12:01.0490990Z self._join_processes(fn) 2025-12-04T09:12:01.0491733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:12:01.0492666Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:12:01.0493452Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:12:01.0494213Z raise RuntimeError(error) 2025-12-04T09:12:01.0494619Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:12:01.0495067Z Traceback (most recent call last): 2025-12-04T09:12:01.0495762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0496465Z getattr(self, test_name)() 2025-12-04T09:12:01.0498638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0499329Z fn() 2025-12-04T09:12:01.0499898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0500578Z method(*args, **kwargs) 2025-12-04T09:12:01.0501216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0501893Z method(*args, **kwargs) 2025-12-04T09:12:01.0502519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0503185Z with policy(): 2025-12-04T09:12:01.0503795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0504467Z raise RuntimeError(msg) 2025-12-04T09:12:01.0505861Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 2. CUDA driver allocated memory was 604962816 and is now 628031488. 2025-12-04T09:12:01.0507036Z 2025-12-04T09:12:01.0507319Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0508398Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0509098Z 2025-12-04T09:12:01.0509385Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0509787Z 2025-12-04T09:12:01.0509793Z 2025-12-04T09:12:01.0510020Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:12:01.0510655Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:12:01.0511895Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-bb030510459950ee.xml - 2025-12-04T09:12:01.0513035Z =========================== short test summary info ============================ 2025-12-04T09:12:01.0514110Z FAILED [8.3006s] distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:12:01.0515125Z Traceback (most recent call last): 2025-12-04T09:12:01.0515923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0516732Z getattr(self, test_name)() 2025-12-04T09:12:01.0517480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0518260Z fn() 2025-12-04T09:12:01.0518997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0519959Z method(*args, **kwargs) 2025-12-04T09:12:01.0520639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0521361Z method(*args, **kwargs) 2025-12-04T09:12:01.0522041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0522744Z with policy(): 2025-12-04T09:12:01.0523391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0524527Z raise RuntimeError(msg) 2025-12-04T09:12:01.0525840Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 2. CUDA driver allocated memory was 604962816 and is now 628031488. 2025-12-04T09:12:01.0527213Z 2025-12-04T09:12:01.0527434Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0528363Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0529058Z 2025-12-04T09:12:01.0529337Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0529932Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:12:01.0530402Z ============================== 1 failed in 8.32s =============================== 2025-12-04T09:12:01.0530809Z Got exit code 1 2025-12-04T09:12:01.0531077Z Retrying single test... 2025-12-04T09:12:01.0531896Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-0f19060d211e471b.xml 2025-12-04T09:12:01.0532846Z ============================= test session starts ============================== 2025-12-04T09:12:01.0533513Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:12:01.0534112Z cachedir: .pytest_cache 2025-12-04T09:12:01.0534808Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:12:01.0535582Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:12:01.0535941Z configfile: pytest.ini 2025-12-04T09:12:01.0536857Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:12:01.0537561Z collecting ... collected 1 item 2025-12-04T09:12:01.0538347Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda 2025-12-04T09:12:01.0539146Z Running 1 items in this shard 2025-12-04T09:12:01.0539333Z 2025-12-04T09:12:01.0540403Z distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda I1204 09:11:33.844000 21264 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 21316 2025-12-04T09:12:01.0541897Z I1204 09:11:33.845000 21264 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 21317 2025-12-04T09:12:01.0542966Z I1204 09:11:33.846000 21264 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 21318 2025-12-04T09:12:01.0544028Z I1204 09:11:33.846000 21264 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 21319 2025-12-04T09:12:01.0545046Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0546114Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0548068Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0549729Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0551373Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0552897Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0554409Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0556097Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0557698Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0559300Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0561024Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0562491Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0563969Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0565474Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0567525Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 1. CUDA driver allocated memory was 604962816 and is now 628031488. 2025-12-04T09:12:01.0569554Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0570600Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0572221Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0573569Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0574655Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0575904Z [rank1]:E1204 09:11:40.649000 21317 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:12:01.0576921Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0577991Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0579484Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0580937Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0582394Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0583751Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0585088Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0586571Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0588267Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0589864Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0591458Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0593021Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0594571Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0596170Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0598376Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 3. CUDA driver allocated memory was 495910912 and is now 628031488. 2025-12-04T09:12:01.0600584Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0601628Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0603235Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0604587Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0605679Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0606935Z [rank3]:E1204 09:11:40.649000 21319 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:12:01.0608029Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0609027Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0610520Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0611980Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0613441Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0614800Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0616185Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0617597Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0619017Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0620426Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0621838Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0623222Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0625210Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0626830Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0629129Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 2. CUDA driver allocated memory was 602865664 and is now 628031488. 2025-12-04T09:12:01.0631201Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0632362Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0634197Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0635729Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0636966Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0638480Z [rank2]:E1204 09:11:40.650000 21318 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:12:01.0639793Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0640805Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0642300Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0643764Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0645215Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0646653Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0647993Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0649418Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0650823Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0652244Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0653675Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0655055Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0656442Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0657850Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0659803Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 714014720 and is now 737083392. 2025-12-04T09:12:01.0661631Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0662676Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0664291Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0665639Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0666782Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0668346Z [rank0]:E1204 09:11:40.652000 21316 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:12:01.0669138Z dist init r=2, world=4 2025-12-04T09:12:01.0669416Z dist init r=3, world=4 2025-12-04T09:12:01.0669702Z dist init r=0, world=4 2025-12-04T09:12:01.0669981Z dist init r=1, world=4 2025-12-04T09:12:01.0671325Z [rank0]:[W1204 09:11:41.993860057 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:12:01.0672723Z FAILED [9.0412s] [100%] 2025-12-04T09:12:01.0672913Z 2025-12-04T09:12:01.0673062Z =================================== FAILURES =================================== 2025-12-04T09:12:01.0673732Z _______________ TestUnevenParamShardCUDA.test_one_iteration_cuda _______________ 2025-12-04T09:12:01.0674279Z Traceback (most recent call last): 2025-12-04T09:12:01.0675059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:12:01.0675856Z self._join_processes(fn) 2025-12-04T09:12:01.0676655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:12:01.0677515Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:12:01.0678401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:12:01.0679263Z raise RuntimeError(error) 2025-12-04T09:12:01.0679821Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:12:01.0680382Z Traceback (most recent call last): 2025-12-04T09:12:01.0681097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0681810Z getattr(self, test_name)() 2025-12-04T09:12:01.0682470Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0683156Z fn() 2025-12-04T09:12:01.0683923Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0684638Z method(*args, **kwargs) 2025-12-04T09:12:01.0685300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0686021Z method(*args, **kwargs) 2025-12-04T09:12:01.0686693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0687392Z with policy(): 2025-12-04T09:12:01.0688042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0688762Z raise RuntimeError(msg) 2025-12-04T09:12:01.0689998Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 714014720 and is now 737083392. 2025-12-04T09:12:01.0691159Z 2025-12-04T09:12:01.0691362Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0692232Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0692908Z 2025-12-04T09:12:01.0693161Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0693538Z 2025-12-04T09:12:01.0693706Z Process 2 exited with error code 10 and exception: 2025-12-04T09:12:01.0694162Z Traceback (most recent call last): 2025-12-04T09:12:01.0694911Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0695665Z getattr(self, test_name)() 2025-12-04T09:12:01.0696374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0697086Z fn() 2025-12-04T09:12:01.0697701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0698416Z method(*args, **kwargs) 2025-12-04T09:12:01.0699073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0699784Z method(*args, **kwargs) 2025-12-04T09:12:01.0700454Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0701227Z with policy(): 2025-12-04T09:12:01.0701864Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0702586Z raise RuntimeError(msg) 2025-12-04T09:12:01.0703915Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 2. CUDA driver allocated memory was 602865664 and is now 628031488. 2025-12-04T09:12:01.0705009Z 2025-12-04T09:12:01.0705214Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0706024Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0706662Z 2025-12-04T09:12:01.0706897Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0707339Z 2025-12-04T09:12:01.0707657Z Process 3 exited with error code 10 and exception: 2025-12-04T09:12:01.0708154Z Traceback (most recent call last): 2025-12-04T09:12:01.0708931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0709728Z getattr(self, test_name)() 2025-12-04T09:12:01.0710484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0711241Z fn() 2025-12-04T09:12:01.0711889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0712647Z method(*args, **kwargs) 2025-12-04T09:12:01.0713355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0714096Z method(*args, **kwargs) 2025-12-04T09:12:01.0714817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0715568Z with policy(): 2025-12-04T09:12:01.0716242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0717012Z raise RuntimeError(msg) 2025-12-04T09:12:01.0718328Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 3. CUDA driver allocated memory was 495910912 and is now 628031488. 2025-12-04T09:12:01.0719560Z 2025-12-04T09:12:01.0719907Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0720726Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0721350Z 2025-12-04T09:12:01.0721656Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0722026Z 2025-12-04T09:12:01.0722030Z 2025-12-04T09:12:01.0722232Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:12:01.0722796Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:12:01.0724219Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-0f19060d211e471b.xml - 2025-12-04T09:12:01.0725352Z =========================== short test summary info ============================ 2025-12-04T09:12:01.0726447Z FAILED [9.0412s] distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:12:01.0727460Z Traceback (most recent call last): 2025-12-04T09:12:01.0728263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0729173Z getattr(self, test_name)() 2025-12-04T09:12:01.0729931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0730704Z fn() 2025-12-04T09:12:01.0731343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0732106Z method(*args, **kwargs) 2025-12-04T09:12:01.0732821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0733588Z method(*args, **kwargs) 2025-12-04T09:12:01.0734286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0735042Z with policy(): 2025-12-04T09:12:01.0735733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0736591Z raise RuntimeError(msg) 2025-12-04T09:12:01.0737755Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 714014720 and is now 737083392. 2025-12-04T09:12:01.0738856Z 2025-12-04T09:12:01.0739049Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0739867Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0740483Z 2025-12-04T09:12:01.0740733Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0741087Z 2025-12-04T09:12:01.0741233Z Process 2 exited with error code 10 and exception: 2025-12-04T09:12:01.0741615Z Traceback (most recent call last): 2025-12-04T09:12:01.0742328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0743033Z getattr(self, test_name)() 2025-12-04T09:12:01.0743706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0744397Z fn() 2025-12-04T09:12:01.0744976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0745646Z method(*args, **kwargs) 2025-12-04T09:12:01.0746282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0746953Z method(*args, **kwargs) 2025-12-04T09:12:01.0747824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0748576Z with policy(): 2025-12-04T09:12:01.0749357Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0750134Z raise RuntimeError(msg) 2025-12-04T09:12:01.0751434Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 2. CUDA driver allocated memory was 602865664 and is now 628031488. 2025-12-04T09:12:01.0752672Z 2025-12-04T09:12:01.0752889Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0753816Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0754517Z 2025-12-04T09:12:01.0754802Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0755203Z 2025-12-04T09:12:01.0755426Z Process 3 exited with error code 10 and exception: 2025-12-04T09:12:01.0755854Z Traceback (most recent call last): 2025-12-04T09:12:01.0756646Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0757448Z getattr(self, test_name)() 2025-12-04T09:12:01.0758186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0758954Z fn() 2025-12-04T09:12:01.0759703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0760505Z method(*args, **kwargs) 2025-12-04T09:12:01.0761143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0761817Z method(*args, **kwargs) 2025-12-04T09:12:01.0762455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0763112Z with policy(): 2025-12-04T09:12:01.0763722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0764403Z raise RuntimeError(msg) 2025-12-04T09:12:01.0765574Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 3. CUDA driver allocated memory was 495910912 and is now 628031488. 2025-12-04T09:12:01.0766668Z 2025-12-04T09:12:01.0766862Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0767680Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0768311Z 2025-12-04T09:12:01.0768549Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0769085Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:12:01.0769505Z ============================== 1 failed in 9.06s =============================== 2025-12-04T09:12:01.0769868Z Got exit code 1 2025-12-04T09:12:01.0770110Z Retrying single test... 2025-12-04T09:12:01.0770840Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-4e3b8ac35e315f80.xml 2025-12-04T09:12:01.0771689Z ============================= test session starts ============================== 2025-12-04T09:12:01.0772277Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:12:01.0772809Z cachedir: .pytest_cache 2025-12-04T09:12:01.0773424Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:12:01.0774116Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:12:01.0774435Z configfile: pytest.ini 2025-12-04T09:12:01.0775149Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:12:01.0775859Z collecting ... collected 1 item 2025-12-04T09:12:01.0776648Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda 2025-12-04T09:12:01.0777445Z Running 1 items in this shard 2025-12-04T09:12:01.0777631Z 2025-12-04T09:12:01.0778490Z distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda I1204 09:11:47.104000 21601 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 21653 2025-12-04T09:12:01.0779903Z I1204 09:11:47.105000 21601 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 21654 2025-12-04T09:12:01.0780922Z I1204 09:11:47.106000 21601 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 21655 2025-12-04T09:12:01.0781992Z I1204 09:11:47.107000 21601 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 21656 2025-12-04T09:12:01.0782949Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0783950Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0785441Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0786912Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0788738Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0790263Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0791773Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0793370Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0794976Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0796582Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0798174Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0799956Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0801353Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0802787Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0804855Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 1. CUDA driver allocated memory was 598671360 and is now 628031488. 2025-12-04T09:12:01.0806677Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0807732Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0809353Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0810714Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0811876Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0813118Z [rank1]:E1204 09:11:53.872000 21654 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:12:01.0814147Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0815155Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0816647Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0818111Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0819573Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0820944Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0822284Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0823856Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0825608Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0827284Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0828893Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0830447Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0832011Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0833721Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0835927Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 2. CUDA driver allocated memory was 604962816 and is now 628031488. 2025-12-04T09:12:01.0837990Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0839381Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0841012Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0842427Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0843519Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0844775Z [rank2]:E1204 09:11:53.873000 21655 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:12:01.0845794Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0846782Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0848277Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0849745Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0851201Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0852565Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0853892Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0855309Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0856927Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0858426Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0859926Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0861380Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0862901Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0864414Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0866704Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 3. CUDA driver allocated memory was 602865664 and is now 628031488. 2025-12-04T09:12:01.0868975Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0870139Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0872038Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0873560Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0874787Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0876184Z [rank3]:E1204 09:11:53.874000 21656 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:12:01.0877327Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:12:01.0878458Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:12:01.0880239Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0881834Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:12:01.0883410Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0884897Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:12:01.0886450Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0887953Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0889567Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0890971Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:12:01.0892386Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0893762Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:12:01.0895231Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0896653Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:12:01.0898586Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 714014720 and is now 737083392. 2025-12-04T09:12:01.0900438Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0901534Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0903150Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0904503Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:12:01.0905582Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0906829Z [rank0]:E1204 09:11:53.875000 21653 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:12:01.0907776Z dist init r=1, world=4 2025-12-04T09:12:01.0908070Z dist init r=2, world=4 2025-12-04T09:12:01.0908349Z dist init r=0, world=4 2025-12-04T09:12:01.0908638Z dist init r=3, world=4 2025-12-04T09:12:01.0909981Z [rank0]:[W1204 09:11:54.218431519 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:12:01.0911354Z FAILED [8.4058s] [100%] 2025-12-04T09:12:01.0911543Z 2025-12-04T09:12:01.0911696Z =================================== FAILURES =================================== 2025-12-04T09:12:01.0912282Z _______________ TestUnevenParamShardCUDA.test_one_iteration_cuda _______________ 2025-12-04T09:12:01.0912829Z Traceback (most recent call last): 2025-12-04T09:12:01.0913609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:12:01.0914411Z self._join_processes(fn) 2025-12-04T09:12:01.0915223Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:12:01.0916081Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:12:01.0916969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:12:01.0917830Z raise RuntimeError(error) 2025-12-04T09:12:01.0918279Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:12:01.0918764Z Traceback (most recent call last): 2025-12-04T09:12:01.0919656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0920508Z getattr(self, test_name)() 2025-12-04T09:12:01.0921167Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0921861Z fn() 2025-12-04T09:12:01.0922505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0923185Z method(*args, **kwargs) 2025-12-04T09:12:01.0924206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0925153Z method(*args, **kwargs) 2025-12-04T09:12:01.0925880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0926637Z with policy(): 2025-12-04T09:12:01.0927312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0928082Z raise RuntimeError(msg) 2025-12-04T09:12:01.0929412Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 714014720 and is now 737083392. 2025-12-04T09:12:01.0930749Z 2025-12-04T09:12:01.0930982Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0931961Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0945190Z 2025-12-04T09:12:01.0945560Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0945932Z 2025-12-04T09:12:01.0945936Z 2025-12-04T09:12:01.0946144Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:12:01.0946715Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:12:01.0948180Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-4e3b8ac35e315f80.xml - 2025-12-04T09:12:01.0949362Z =========================== short test summary info ============================ 2025-12-04T09:12:01.0950438Z FAILED [8.4058s] distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:12:01.0951454Z Traceback (most recent call last): 2025-12-04T09:12:01.0952255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:12:01.0953063Z getattr(self, test_name)() 2025-12-04T09:12:01.0953813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:12:01.0954588Z fn() 2025-12-04T09:12:01.0955245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0955997Z method(*args, **kwargs) 2025-12-04T09:12:01.0956718Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:12:01.0957482Z method(*args, **kwargs) 2025-12-04T09:12:01.0958201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:12:01.0958945Z with policy(): 2025-12-04T09:12:01.0959737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:12:01.0960560Z raise RuntimeError(msg) 2025-12-04T09:12:01.0961723Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestUnevenParamShardCUDA.test_one_iteration_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 714014720 and is now 737083392. 2025-12-04T09:12:01.0962841Z 2025-12-04T09:12:01.0963034Z To execute this test, run the following from the base repo dir: 2025-12-04T09:12:01.0964020Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_uneven.py TestUnevenParamShardCUDA.test_one_iteration_cuda 2025-12-04T09:12:01.0964648Z 2025-12-04T09:12:01.0964902Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:12:01.0965431Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:12:01.0965850Z ============================== 1 failed in 8.43s =============================== 2025-12-04T09:12:01.0966213Z Got exit code 1 2025-12-04T09:12:01.0966800Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda 2025-12-04T09:12:01.0967712Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:12:01.0968769Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-7270531c264bcf9a.xml 2025-12-04T09:12:01.0969615Z ============================= test session starts ============================== 2025-12-04T09:12:01.0970263Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:12:01.0970805Z cachedir: .pytest_cache 2025-12-04T09:12:01.0971435Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:12:01.0972116Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:12:01.0972429Z configfile: pytest.ini 2025-12-04T09:12:01.0973075Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:12:01.0973839Z collecting ... collected 1 item / 1 deselected / 0 selected 2025-12-04T09:12:01.0974264Z stepcurrent: skipping 1 already run items. 2025-12-04T09:12:01.0974606Z Running 0 items in this shard 2025-12-04T09:12:01.0974791Z 2025-12-04T09:12:01.0975532Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-7270531c264bcf9a.xml - 2025-12-04T09:12:01.0976524Z ============================ 1 deselected in 0.01s ============================= 2025-12-04T09:12:01.0977332Z The following tests failed consistently: ['test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda'] 2025-12-04T09:12:01.0977983Z 2025-12-04T09:12:01.0978545Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven 1/1 (test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_04d990d4b348b53e_.log) 2025-12-04T09:12:01.0979222Z 2025-12-04T09:12:01.0979584Z Finished distributed/fsdp/test_fsdp_uneven 1/1 ... [2025-12-04 09:12:01.033956][1151.971590424], took 0.73min 2025-12-04T09:12:01.0980820Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-bb030510459950ee.xml 2025-12-04T09:12:01.1164291Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-0f19060d211e471b.xml 2025-12-04T09:12:01.1471304Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-4e3b8ac35e315f80.xml 2025-12-04T09:12:01.1782651Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-7270531c264bcf9a.xml 2025-12-04T09:12:01.4002247Z Uploading logs for 57116084912 to S3 2025-12-04T09:12:01.4288887Z Uploading artifacts took 0.22 seconds 2025-12-04T09:12:01.4289347Z distributed/fsdp/test_fsdp_uneven 1/1 failed! 2025-12-04T09:12:01.4291692Z Running distributed/fsdp/test_fsdp_input 1/1 ... [2025-12-04 09:12:01.429045][1152.366681348] 2025-12-04T09:12:01.4292268Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:12:01.4295654Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:12:01.429360] 2025-12-04T09:13:07.7956812Z 2025-12-04T09:13:07.7957895Z PRINTING LOG FILE of distributed/fsdp/test_fsdp_input 1/1 (test/test-reports/distributed.fsdp.test_fsdp_input_1.1_b656a1fd46e2d098_.log) 2025-12-04T09:13:07.7959259Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-ed7a8589598da4ef.xml 2025-12-04T09:13:07.7960192Z ============================= test session starts ============================== 2025-12-04T09:13:07.7960854Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:07.7961453Z cachedir: .pytest_cache 2025-12-04T09:13:07.7962173Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:07.7963295Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:07.7963655Z configfile: pytest.ini 2025-12-04T09:13:07.7964536Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:07.7965301Z collecting ... collected 2 items 2025-12-04T09:13:07.7965700Z stepcurrent: Cannot find last run test, not skipping 2025-12-04T09:13:07.7966864Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda, test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda 2025-12-04T09:13:07.7967826Z 2025-12-04T09:13:07.7968697Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda I1204 09:12:04.874000 21995 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 22047 2025-12-04T09:13:07.7970708Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2025-12-04T09:13:07.7971950Z _init_core_state( 2025-12-04T09:13:07.7976460Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:13:07.7981349Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:13:07.7982330Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:07.7983420Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:07.7985054Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.7986818Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:07.7988754Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.7990288Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:07.7991788Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.7993390Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.7995059Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.7996688Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.7998238Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.7999737Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:07.8001252Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8002822Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:07.8004888Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8006812Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8007939Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8009651Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8011079Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8012272Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8013645Z [rank0]:E1204 09:12:09.238000 22047 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:13:07.8014406Z dist init r=0, world=1 2025-12-04T09:13:07.8015719Z [rank0]:[W1204 09:12:09.479170943 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:13:07.8017079Z FAILED [6.3111s] [ 50%] 2025-12-04T09:13:07.8017252Z 2025-12-04T09:13:07.8017476Z =================================== FAILURES =================================== 2025-12-04T09:13:07.8018004Z ___________________ TestInputCUDA.test_input_type_dict_cuda ____________________ 2025-12-04T09:13:07.8018504Z Traceback (most recent call last): 2025-12-04T09:13:07.8019274Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:07.8020037Z self._join_processes(fn) 2025-12-04T09:13:07.8020822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:07.8021672Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:07.8022534Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:07.8023367Z raise RuntimeError(error) 2025-12-04T09:13:07.8024202Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8024818Z Traceback (most recent call last): 2025-12-04T09:13:07.8025594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8026408Z getattr(self, test_name)() 2025-12-04T09:13:07.8027163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8028016Z fn() 2025-12-04T09:13:07.8028653Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8029410Z method(*args, **kwargs) 2025-12-04T09:13:07.8030128Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8030873Z method(*args, **kwargs) 2025-12-04T09:13:07.8031590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8032357Z with policy(): 2025-12-04T09:13:07.8033138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8033853Z raise RuntimeError(msg) 2025-12-04T09:13:07.8035031Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8036143Z 2025-12-04T09:13:07.8036346Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8037161Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8037767Z 2025-12-04T09:13:07.8038022Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8038420Z 2025-12-04T09:13:07.8038425Z 2025-12-04T09:13:07.8038644Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:07.8039242Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:07.8040393Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-ed7a8589598da4ef.xml - 2025-12-04T09:13:07.8041442Z =========================== short test summary info ============================ 2025-12-04T09:13:07.8042381Z FAILED [6.3111s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8043258Z Traceback (most recent call last): 2025-12-04T09:13:07.8044180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8044952Z getattr(self, test_name)() 2025-12-04T09:13:07.8045785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8046545Z fn() 2025-12-04T09:13:07.8047179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8047911Z method(*args, **kwargs) 2025-12-04T09:13:07.8048613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8049354Z method(*args, **kwargs) 2025-12-04T09:13:07.8050032Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8050767Z with policy(): 2025-12-04T09:13:07.8051432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8052231Z raise RuntimeError(msg) 2025-12-04T09:13:07.8053435Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8054576Z 2025-12-04T09:13:07.8054787Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8055626Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8056249Z 2025-12-04T09:13:07.8056518Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8057079Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:07.8057545Z ============================== 1 failed in 6.33s =============================== 2025-12-04T09:13:07.8057932Z Got exit code 1 2025-12-04T09:13:07.8058185Z Retrying single test... 2025-12-04T09:13:07.8058999Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-68059153dd3ae49c.xml 2025-12-04T09:13:07.8059910Z ============================= test session starts ============================== 2025-12-04T09:13:07.8060551Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:07.8061119Z cachedir: .pytest_cache 2025-12-04T09:13:07.8061803Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:07.8062562Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:07.8062905Z configfile: pytest.ini 2025-12-04T09:13:07.8063688Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:07.8064521Z collecting ... collected 2 items / 1 deselected / 1 selected 2025-12-04T09:13:07.8065410Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda 2025-12-04T09:13:07.8066190Z Running 1 items in this shard 2025-12-04T09:13:07.8066400Z 2025-12-04T09:13:07.8067296Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda I1204 09:12:15.234000 22118 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 22170 2025-12-04T09:13:07.8069493Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2025-12-04T09:13:07.8070767Z _init_core_state( 2025-12-04T09:13:07.8075533Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:13:07.8080689Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:13:07.8081578Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:07.8082634Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:07.8084130Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8085599Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:07.8087057Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8088423Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:07.8089759Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8091182Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8092783Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8094281Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8095787Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8097246Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:07.8098713Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8100221Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:07.8102224Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8104329Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8105454Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8107171Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8108866Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8110096Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8111496Z [rank0]:E1204 09:12:19.559000 22170 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:13:07.8112363Z dist init r=0, world=1 2025-12-04T09:13:07.8113708Z [rank0]:[W1204 09:12:19.803334467 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:13:07.8115099Z FAILED [6.2754s] [100%] 2025-12-04T09:13:07.8115275Z 2025-12-04T09:13:07.8115427Z =================================== FAILURES =================================== 2025-12-04T09:13:07.8115981Z ___________________ TestInputCUDA.test_input_type_dict_cuda ____________________ 2025-12-04T09:13:07.8116495Z Traceback (most recent call last): 2025-12-04T09:13:07.8117285Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:07.8118067Z self._join_processes(fn) 2025-12-04T09:13:07.8118878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:07.8119857Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:07.8120814Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:07.8121620Z raise RuntimeError(error) 2025-12-04T09:13:07.8122046Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8122517Z Traceback (most recent call last): 2025-12-04T09:13:07.8123243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8124158Z getattr(self, test_name)() 2025-12-04T09:13:07.8125093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8125880Z fn() 2025-12-04T09:13:07.8126537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8127306Z method(*args, **kwargs) 2025-12-04T09:13:07.8128022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8128769Z method(*args, **kwargs) 2025-12-04T09:13:07.8129484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8130233Z with policy(): 2025-12-04T09:13:07.8130922Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8131681Z raise RuntimeError(msg) 2025-12-04T09:13:07.8133051Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8134226Z 2025-12-04T09:13:07.8134462Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8135332Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8135975Z 2025-12-04T09:13:07.8136242Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8136767Z 2025-12-04T09:13:07.8136771Z 2025-12-04T09:13:07.8136973Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:07.8137535Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:07.8138618Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-68059153dd3ae49c.xml - 2025-12-04T09:13:07.8139605Z =========================== short test summary info ============================ 2025-12-04T09:13:07.8140782Z FAILED [6.2754s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8141667Z Traceback (most recent call last): 2025-12-04T09:13:07.8142412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8143156Z getattr(self, test_name)() 2025-12-04T09:13:07.8143868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8144597Z fn() 2025-12-04T09:13:07.8145203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8145920Z method(*args, **kwargs) 2025-12-04T09:13:07.8146591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8147381Z method(*args, **kwargs) 2025-12-04T09:13:07.8148245Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8148996Z with policy(): 2025-12-04T09:13:07.8149686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8150445Z raise RuntimeError(msg) 2025-12-04T09:13:07.8151696Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8152870Z 2025-12-04T09:13:07.8153086Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8153952Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8154600Z 2025-12-04T09:13:07.8154876Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8155454Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:07.8155959Z ======================= 1 failed, 1 deselected in 6.30s ======================== 2025-12-04T09:13:07.8156381Z Got exit code 1 2025-12-04T09:13:07.8156637Z Retrying single test... 2025-12-04T09:13:07.8157467Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-dd8d334d3533ae96.xml 2025-12-04T09:13:07.8158411Z ============================= test session starts ============================== 2025-12-04T09:13:07.8159072Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:07.8159765Z cachedir: .pytest_cache 2025-12-04T09:13:07.8160592Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:07.8161287Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:07.8161592Z configfile: pytest.ini 2025-12-04T09:13:07.8162417Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:07.8163253Z collecting ... collected 2 items / 1 deselected / 1 selected 2025-12-04T09:13:07.8164142Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda 2025-12-04T09:13:07.8164921Z Running 1 items in this shard 2025-12-04T09:13:07.8165134Z 2025-12-04T09:13:07.8165963Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda I1204 09:12:25.584000 22241 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 22293 2025-12-04T09:13:07.8167962Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2025-12-04T09:13:07.8169153Z _init_core_state( 2025-12-04T09:13:07.8173532Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:13:07.8178228Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:13:07.8179171Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:07.8180456Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:07.8181950Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8183431Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:07.8184886Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8186232Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:07.8187804Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8189405Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8191062Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8192668Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8194248Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8195798Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:07.8197360Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8199021Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:07.8201132Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8202884Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8203925Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8205496Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8206819Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8207917Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8209151Z [rank0]:E1204 09:12:29.922000 22293 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:13:07.8209859Z dist init r=0, world=1 2025-12-04T09:13:07.8211057Z [rank0]:[W1204 09:12:30.169827519 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:13:07.8212286Z FAILED [6.2847s] [100%] 2025-12-04T09:13:07.8212445Z 2025-12-04T09:13:07.8212587Z =================================== FAILURES =================================== 2025-12-04T09:13:07.8213079Z ___________________ TestInputCUDA.test_input_type_dict_cuda ____________________ 2025-12-04T09:13:07.8213541Z Traceback (most recent call last): 2025-12-04T09:13:07.8214242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:07.8214939Z self._join_processes(fn) 2025-12-04T09:13:07.8215651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:07.8216425Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:07.8217230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:07.8218003Z raise RuntimeError(error) 2025-12-04T09:13:07.8218399Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8218900Z Traceback (most recent call last): 2025-12-04T09:13:07.8219598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8220309Z getattr(self, test_name)() 2025-12-04T09:13:07.8220971Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8221660Z fn() 2025-12-04T09:13:07.8222241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8222915Z method(*args, **kwargs) 2025-12-04T09:13:07.8223689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8224720Z method(*args, **kwargs) 2025-12-04T09:13:07.8225438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8226292Z with policy(): 2025-12-04T09:13:07.8226984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8227825Z raise RuntimeError(msg) 2025-12-04T09:13:07.8229058Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8230238Z 2025-12-04T09:13:07.8230455Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8231321Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8231964Z 2025-12-04T09:13:07.8232245Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8232651Z 2025-12-04T09:13:07.8232656Z 2025-12-04T09:13:07.8232896Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:07.8233518Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:07.8234738Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-dd8d334d3533ae96.xml - 2025-12-04T09:13:07.8235869Z =========================== short test summary info ============================ 2025-12-04T09:13:07.8236869Z FAILED [6.2847s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8237793Z Traceback (most recent call last): 2025-12-04T09:13:07.8238586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8239487Z getattr(self, test_name)() 2025-12-04T09:13:07.8240151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8240842Z fn() 2025-12-04T09:13:07.8241417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8242100Z method(*args, **kwargs) 2025-12-04T09:13:07.8242723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8243403Z method(*args, **kwargs) 2025-12-04T09:13:07.8244037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8244872Z with policy(): 2025-12-04T09:13:07.8245516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8246242Z raise RuntimeError(msg) 2025-12-04T09:13:07.8247495Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_dict_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8248589Z 2025-12-04T09:13:07.8248795Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8249607Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_dict_cuda 2025-12-04T09:13:07.8250223Z 2025-12-04T09:13:07.8250474Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8251030Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:07.8251495Z ======================= 1 failed, 1 deselected in 6.31s ======================== 2025-12-04T09:13:07.8251894Z Got exit code 1 2025-12-04T09:13:07.8252518Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda 2025-12-04T09:13:07.8253438Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:13:07.8254547Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-571ae876e059f56e.xml 2025-12-04T09:13:07.8255440Z ============================= test session starts ============================== 2025-12-04T09:13:07.8256064Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:07.8256637Z cachedir: .pytest_cache 2025-12-04T09:13:07.8257408Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:07.8258100Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:07.8258417Z configfile: pytest.ini 2025-12-04T09:13:07.8259057Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:07.8259848Z collecting ... collected 2 items / 1 deselected / 1 selected 2025-12-04T09:13:07.8260281Z stepcurrent: skipping 1 already run items. 2025-12-04T09:13:07.8260627Z Running 1 items in this shard 2025-12-04T09:13:07.8260811Z 2025-12-04T09:13:07.8261604Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda I1204 09:12:36.004000 22364 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 22416 2025-12-04T09:13:07.8263421Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2025-12-04T09:13:07.8264538Z _init_core_state( 2025-12-04T09:13:07.8269055Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:13:07.8274073Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:13:07.8275145Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:07.8276270Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:07.8277954Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8279716Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:07.8281320Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8282762Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:07.8284089Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8285503Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8286915Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8288327Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8289750Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8291122Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:07.8292507Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8293927Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:07.8295821Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8297589Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8298620Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8300185Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8301497Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8302587Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8303886Z [rank0]:E1204 09:12:40.309000 22416 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:13:07.8304583Z dist init r=0, world=1 2025-12-04T09:13:07.8305781Z [rank0]:[W1204 09:12:40.553493442 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:13:07.8307016Z FAILED [6.3607s] [100%] 2025-12-04T09:13:07.8307174Z 2025-12-04T09:13:07.8307385Z =================================== FAILURES =================================== 2025-12-04T09:13:07.8308063Z ___________________ TestInputCUDA.test_input_type_list_cuda ____________________ 2025-12-04T09:13:07.8308581Z Traceback (most recent call last): 2025-12-04T09:13:07.8309372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:07.8310237Z self._join_processes(fn) 2025-12-04T09:13:07.8311043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:07.8311916Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:07.8312805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:07.8313664Z raise RuntimeError(error) 2025-12-04T09:13:07.8314117Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8314615Z Traceback (most recent call last): 2025-12-04T09:13:07.8315386Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8316188Z getattr(self, test_name)() 2025-12-04T09:13:07.8316945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8317731Z fn() 2025-12-04T09:13:07.8318370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8319137Z method(*args, **kwargs) 2025-12-04T09:13:07.8319959Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8320753Z method(*args, **kwargs) 2025-12-04T09:13:07.8321396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8322072Z with policy(): 2025-12-04T09:13:07.8322685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8323364Z raise RuntimeError(msg) 2025-12-04T09:13:07.8325036Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8326218Z 2025-12-04T09:13:07.8326439Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8327304Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8327948Z 2025-12-04T09:13:07.8328216Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8328634Z 2025-12-04T09:13:07.8328639Z 2025-12-04T09:13:07.8328861Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:07.8329489Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:07.8330717Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-571ae876e059f56e.xml - 2025-12-04T09:13:07.8331970Z =========================== short test summary info ============================ 2025-12-04T09:13:07.8332972Z FAILED [6.3607s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8333908Z Traceback (most recent call last): 2025-12-04T09:13:07.8334703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8335494Z getattr(self, test_name)() 2025-12-04T09:13:07.8336354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8337189Z fn() 2025-12-04T09:13:07.8337770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8338506Z method(*args, **kwargs) 2025-12-04T09:13:07.8339147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8339830Z method(*args, **kwargs) 2025-12-04T09:13:07.8340448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8341120Z with policy(): 2025-12-04T09:13:07.8341733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8342419Z raise RuntimeError(msg) 2025-12-04T09:13:07.8343510Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8344550Z 2025-12-04T09:13:07.8344741Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8345513Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8346085Z 2025-12-04T09:13:07.8346335Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8346847Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:07.8347355Z ======================= 1 failed, 1 deselected in 6.38s ======================== 2025-12-04T09:13:07.8347916Z Got exit code 1 2025-12-04T09:13:07.8348180Z Retrying single test... 2025-12-04T09:13:07.8349018Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-6f9ee7a291766819.xml 2025-12-04T09:13:07.8349964Z ============================= test session starts ============================== 2025-12-04T09:13:07.8350630Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:07.8351228Z cachedir: .pytest_cache 2025-12-04T09:13:07.8351935Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:07.8352716Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:07.8353069Z configfile: pytest.ini 2025-12-04T09:13:07.8353781Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:07.8354667Z collecting ... collected 2 items / 1 deselected / 1 selected 2025-12-04T09:13:07.8357128Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda 2025-12-04T09:13:07.8357966Z Running 1 items in this shard 2025-12-04T09:13:07.8358194Z 2025-12-04T09:13:07.8359192Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda I1204 09:12:46.384000 22487 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 22539 2025-12-04T09:13:07.8361331Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2025-12-04T09:13:07.8362463Z _init_core_state( 2025-12-04T09:13:07.8366613Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:13:07.8371124Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:13:07.8372018Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:07.8373015Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:07.8374522Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8375987Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:07.8377453Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8378822Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:07.8380148Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8381564Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8382988Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8384405Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8385804Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8387183Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:07.8389037Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8390653Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:07.8392782Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8394782Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8395940Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8397714Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8399248Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8400536Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8401776Z [rank0]:E1204 09:12:50.708000 22539 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:13:07.8402489Z dist init r=0, world=1 2025-12-04T09:13:07.8403686Z [rank0]:[W1204 09:12:51.951693197 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:13:07.8404928Z FAILED [6.3024s] [100%] 2025-12-04T09:13:07.8405090Z 2025-12-04T09:13:07.8405224Z =================================== FAILURES =================================== 2025-12-04T09:13:07.8405714Z ___________________ TestInputCUDA.test_input_type_list_cuda ____________________ 2025-12-04T09:13:07.8406178Z Traceback (most recent call last): 2025-12-04T09:13:07.8406882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:07.8407583Z self._join_processes(fn) 2025-12-04T09:13:07.8408301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:07.8409081Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:07.8410050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:07.8410871Z raise RuntimeError(error) 2025-12-04T09:13:07.8411305Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8411777Z Traceback (most recent call last): 2025-12-04T09:13:07.8412505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8413260Z getattr(self, test_name)() 2025-12-04T09:13:07.8413983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8414721Z fn() 2025-12-04T09:13:07.8415323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8416044Z method(*args, **kwargs) 2025-12-04T09:13:07.8416717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8417418Z method(*args, **kwargs) 2025-12-04T09:13:07.8418156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8418873Z with policy(): 2025-12-04T09:13:07.8419517Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8420227Z raise RuntimeError(msg) 2025-12-04T09:13:07.8421402Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8422498Z 2025-12-04T09:13:07.8422715Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8423531Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8424732Z 2025-12-04T09:13:07.8425005Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8425424Z 2025-12-04T09:13:07.8425429Z 2025-12-04T09:13:07.8425654Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:07.8426288Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:07.8427591Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-6f9ee7a291766819.xml - 2025-12-04T09:13:07.8428703Z =========================== short test summary info ============================ 2025-12-04T09:13:07.8429709Z FAILED [6.3024s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8430648Z Traceback (most recent call last): 2025-12-04T09:13:07.8431441Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8432241Z getattr(self, test_name)() 2025-12-04T09:13:07.8432999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8433773Z fn() 2025-12-04T09:13:07.8434412Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8435179Z method(*args, **kwargs) 2025-12-04T09:13:07.8435894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8436651Z method(*args, **kwargs) 2025-12-04T09:13:07.8437350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8438106Z with policy(): 2025-12-04T09:13:07.8438797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8439743Z raise RuntimeError(msg) 2025-12-04T09:13:07.8440859Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8441905Z 2025-12-04T09:13:07.8442099Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8442874Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8443443Z 2025-12-04T09:13:07.8443689Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8444203Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:07.8444654Z ======================= 1 failed, 1 deselected in 6.32s ======================== 2025-12-04T09:13:07.8445035Z Got exit code 1 2025-12-04T09:13:07.8445375Z Retrying single test... 2025-12-04T09:13:07.8446114Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-7369439069792ace.xml 2025-12-04T09:13:07.8446952Z ============================= test session starts ============================== 2025-12-04T09:13:07.8447547Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:07.8448076Z cachedir: .pytest_cache 2025-12-04T09:13:07.8448708Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:07.8449407Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:07.8449710Z configfile: pytest.ini 2025-12-04T09:13:07.8450352Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:07.8451209Z collecting ... collected 2 items / 1 deselected / 1 selected 2025-12-04T09:13:07.8452043Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda 2025-12-04T09:13:07.8452775Z Running 1 items in this shard 2025-12-04T09:13:07.8452974Z 2025-12-04T09:13:07.8453767Z distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda I1204 09:12:56.774000 22610 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 22662 2025-12-04T09:13:07.8455584Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:479: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1. 2025-12-04T09:13:07.8456710Z _init_core_state( 2025-12-04T09:13:07.8460823Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:13:07.8465273Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:13:07.8466170Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:07.8467424Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:07.8469251Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8470919Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:07.8472567Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8474150Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:07.8475657Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8477260Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8478850Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8480757Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:07.8482166Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8483614Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:07.8484994Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8486422Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:07.8488310Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8490071Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8491120Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8492692Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8493999Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:07.8495092Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8496333Z [rank0]:E1204 09:13:01.099000 22662 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:13:07.8497043Z dist init r=0, world=1 2025-12-04T09:13:07.8498244Z [rank0]:[W1204 09:13:01.344494310 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:13:07.8499483Z FAILED [6.2983s] [100%] 2025-12-04T09:13:07.8499640Z 2025-12-04T09:13:07.8499777Z =================================== FAILURES =================================== 2025-12-04T09:13:07.8500268Z ___________________ TestInputCUDA.test_input_type_list_cuda ____________________ 2025-12-04T09:13:07.8500728Z Traceback (most recent call last): 2025-12-04T09:13:07.8501433Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:07.8502135Z self._join_processes(fn) 2025-12-04T09:13:07.8502902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:07.8503680Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:07.8504460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:07.8505234Z raise RuntimeError(error) 2025-12-04T09:13:07.8505642Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8506087Z Traceback (most recent call last): 2025-12-04T09:13:07.8506774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8507562Z getattr(self, test_name)() 2025-12-04T09:13:07.8508485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8509322Z fn() 2025-12-04T09:13:07.8509979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8510740Z method(*args, **kwargs) 2025-12-04T09:13:07.8511462Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8512213Z method(*args, **kwargs) 2025-12-04T09:13:07.8512932Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8513685Z with policy(): 2025-12-04T09:13:07.8514355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8515122Z raise RuntimeError(msg) 2025-12-04T09:13:07.8516376Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8517559Z 2025-12-04T09:13:07.8517790Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8518639Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8519292Z 2025-12-04T09:13:07.8519561Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8520085Z 2025-12-04T09:13:07.8520089Z 2025-12-04T09:13:07.8520288Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:07.8520841Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:07.8521917Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-7369439069792ace.xml - 2025-12-04T09:13:07.8522904Z =========================== short test summary info ============================ 2025-12-04T09:13:07.8524103Z FAILED [6.2983s] distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:13:07.8525233Z Traceback (most recent call last): 2025-12-04T09:13:07.8526015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:07.8526824Z getattr(self, test_name)() 2025-12-04T09:13:07.8527591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:07.8528361Z fn() 2025-12-04T09:13:07.8528998Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8529765Z method(*args, **kwargs) 2025-12-04T09:13:07.8530612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:07.8531378Z method(*args, **kwargs) 2025-12-04T09:13:07.8532080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:07.8532833Z with policy(): 2025-12-04T09:13:07.8533515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:07.8534283Z raise RuntimeError(msg) 2025-12-04T09:13:07.8535529Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestInputCUDA.test_input_type_list_cuda! Caching allocator allocated memory was 512 and is now reported as 1024 on device 0. CUDA driver allocated memory was 713949184 and is now 737017856. 2025-12-04T09:13:07.8536811Z 2025-12-04T09:13:07.8537003Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:07.8537846Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_input.py TestInputCUDA.test_input_type_list_cuda 2025-12-04T09:13:07.8538415Z 2025-12-04T09:13:07.8538667Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:07.8539178Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:07.8539631Z ======================= 1 failed, 1 deselected in 6.32s ======================== 2025-12-04T09:13:07.8540013Z Got exit code 1 2025-12-04T09:13:07.8540541Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda 2025-12-04T09:13:07.8541413Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:13:07.8542466Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-f5da965ca571529c.xml 2025-12-04T09:13:07.8543314Z ============================= test session starts ============================== 2025-12-04T09:13:07.8543891Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:07.8544429Z cachedir: .pytest_cache 2025-12-04T09:13:07.8545053Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:07.8545746Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:07.8546049Z configfile: pytest.ini 2025-12-04T09:13:07.8546697Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:07.8547551Z collecting ... collected 2 items / 2 deselected / 0 selected 2025-12-04T09:13:07.8548185Z stepcurrent: skipping 2 already run items. 2025-12-04T09:13:07.8548576Z Running 0 items in this shard 2025-12-04T09:13:07.8548799Z 2025-12-04T09:13:07.8549623Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-f5da965ca571529c.xml - 2025-12-04T09:13:07.8550742Z ============================ 2 deselected in 0.01s ============================= 2025-12-04T09:13:07.8551995Z The following tests failed consistently: ['test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda', 'test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda'] 2025-12-04T09:13:07.8553081Z 2025-12-04T09:13:07.8553710Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input 1/1 (test/test-reports/distributed.fsdp.test_fsdp_input_1.1_b656a1fd46e2d098_.log) 2025-12-04T09:13:07.8554479Z 2025-12-04T09:13:07.8554863Z Finished distributed/fsdp/test_fsdp_input 1/1 ... [2025-12-04 09:13:07.796402][1218.734035176], took 1.11min 2025-12-04T09:13:07.8556254Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-ed7a8589598da4ef.xml 2025-12-04T09:13:07.8720422Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-68059153dd3ae49c.xml 2025-12-04T09:13:07.9025811Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-dd8d334d3533ae96.xml 2025-12-04T09:13:07.9303533Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-571ae876e059f56e.xml 2025-12-04T09:13:07.9657813Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-6f9ee7a291766819.xml 2025-12-04T09:13:07.9951528Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-7369439069792ace.xml 2025-12-04T09:13:08.0265713Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-f5da965ca571529c.xml 2025-12-04T09:13:08.2501943Z Uploading logs for 57116084912 to S3 2025-12-04T09:13:08.2799579Z Uploading artifacts took 0.24 seconds 2025-12-04T09:13:08.2800122Z distributed/fsdp/test_fsdp_input 1/1 failed! 2025-12-04T09:13:08.2805566Z Running distributed/fsdp/test_fsdp_traversal 1/1 ... [2025-12-04 09:13:08.279912][1219.217548727] 2025-12-04T09:13:08.2806186Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:13:08.2807464Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:13:08.280227] 2025-12-04T09:13:45.2791636Z 2025-12-04T09:13:45.2792647Z PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal 1/1 (test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_e348e090749a4a32_.log) 2025-12-04T09:13:45.2794117Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-551e66d027b33cdd.xml 2025-12-04T09:13:45.2795118Z ============================= test session starts ============================== 2025-12-04T09:13:45.2795792Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:45.2796401Z cachedir: .pytest_cache 2025-12-04T09:13:45.2797096Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:45.2797959Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:45.2798316Z configfile: pytest.ini 2025-12-04T09:13:45.2799036Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:45.2799960Z collecting ... collected 1 item 2025-12-04T09:13:45.2800355Z stepcurrent: Cannot find last run test, not skipping 2025-12-04T09:13:45.2801150Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda 2025-12-04T09:13:45.2801738Z 2025-12-04T09:13:45.2802635Z distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda I1204 09:13:11.704000 22790 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 22842 2025-12-04T09:13:45.2804143Z I1204 09:13:11.705000 22790 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 22843 2025-12-04T09:13:45.2806714Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:3328: UserWarning: CUDA caching allocator reports a memory leak not verified by the driver API in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 0. CUDA driver allocated memory was 456065024 and is now 456065024. 2025-12-04T09:13:45.2808664Z with policy(): 2025-12-04T09:13:45.2809274Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:45.2810368Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:45.2812004Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.2813608Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:45.2815337Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.2816831Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:45.2818282Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2819831Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:45.2821377Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2822931Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:45.2824931Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.2826494Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:45.2828158Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.2829774Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:45.2831937Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.2833970Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:45.2835132Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.2836922Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.2838419Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:45.2839843Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.2841205Z [rank1]:E1204 09:13:16.638000 22843 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:13:45.2841977Z dist init r=1, world=2 2025-12-04T09:13:45.2842260Z FAILED [6.6942s] [100%] 2025-12-04T09:13:45.2842434Z 2025-12-04T09:13:45.2842595Z =================================== FAILURES =================================== 2025-12-04T09:13:45.2843124Z ___________________ TestTraversalCUDA.test_fsdp_modules_cuda ___________________ 2025-12-04T09:13:45.2843631Z Traceback (most recent call last): 2025-12-04T09:13:45.2844392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:45.2845240Z self._join_processes(fn) 2025-12-04T09:13:45.2846020Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:45.2846863Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:45.2847723Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:45.2848549Z raise RuntimeError(error) 2025-12-04T09:13:45.2848992Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:13:45.2849479Z Traceback (most recent call last): 2025-12-04T09:13:45.2850227Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.2851001Z getattr(self, test_name)() 2025-12-04T09:13:45.2851734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.2852493Z fn() 2025-12-04T09:13:45.2853113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2853851Z method(*args, **kwargs) 2025-12-04T09:13:45.2854542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2855266Z method(*args, **kwargs) 2025-12-04T09:13:45.2855953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.2856685Z with policy(): 2025-12-04T09:13:45.2857346Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.2858083Z raise RuntimeError(msg) 2025-12-04T09:13:45.2859306Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.2860465Z 2025-12-04T09:13:45.2860676Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.2861538Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.2862192Z 2025-12-04T09:13:45.2862449Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.2862852Z 2025-12-04T09:13:45.2862857Z 2025-12-04T09:13:45.2863076Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:45.2863688Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:45.2864916Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-551e66d027b33cdd.xml - 2025-12-04T09:13:45.2866101Z =========================== short test summary info ============================ 2025-12-04T09:13:45.2867115Z FAILED [6.6942s] distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:13:45.2868315Z Traceback (most recent call last): 2025-12-04T09:13:45.2869111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.2869899Z getattr(self, test_name)() 2025-12-04T09:13:45.2870654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.2871427Z fn() 2025-12-04T09:13:45.2872084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2872834Z method(*args, **kwargs) 2025-12-04T09:13:45.2873619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2874389Z method(*args, **kwargs) 2025-12-04T09:13:45.2875088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.2875838Z with policy(): 2025-12-04T09:13:45.2876526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.2877466Z raise RuntimeError(msg) 2025-12-04T09:13:45.2878790Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.2880265Z 2025-12-04T09:13:45.2880531Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.2881529Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.2882314Z 2025-12-04T09:13:45.2882625Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.2883342Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:45.2883870Z ============================== 1 failed in 6.91s =============================== 2025-12-04T09:13:45.2884333Z Got exit code 1 2025-12-04T09:13:45.2894693Z Retrying single test... 2025-12-04T09:13:45.2895603Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-8583a5ccabf54360.xml 2025-12-04T09:13:45.2896532Z ============================= test session starts ============================== 2025-12-04T09:13:45.2897174Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:45.2897763Z cachedir: .pytest_cache 2025-12-04T09:13:45.2898423Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:45.2899165Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:45.2899502Z configfile: pytest.ini 2025-12-04T09:13:45.2900189Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:45.2900925Z collecting ... collected 1 item 2025-12-04T09:13:45.2901740Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda 2025-12-04T09:13:45.2902556Z Running 1 items in this shard 2025-12-04T09:13:45.2902758Z 2025-12-04T09:13:45.2903642Z distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda I1204 09:13:22.844000 22972 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 23024 2025-12-04T09:13:45.2905231Z I1204 09:13:22.845000 22972 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 23025 2025-12-04T09:13:45.2907588Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:3328: UserWarning: CUDA caching allocator reports a memory leak not verified by the driver API in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 0. CUDA driver allocated memory was 456065024 and is now 456065024. 2025-12-04T09:13:45.2909741Z with policy(): 2025-12-04T09:13:45.2910372Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:45.2911496Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:45.2913269Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.2914927Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:45.2916573Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.2918106Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:45.2919710Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2921288Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:45.2922710Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2924518Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:45.2926282Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.2927827Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:45.2929394Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.2931000Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:45.2933125Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.2935132Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:45.2936303Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.2938281Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.2939679Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:45.2940842Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.2942259Z [rank1]:E1204 09:13:27.861000 23025 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:13:45.2942970Z dist init r=1, world=2 2025-12-04T09:13:45.2943219Z FAILED [6.7291s] [100%] 2025-12-04T09:13:45.2943391Z 2025-12-04T09:13:45.2943529Z =================================== FAILURES =================================== 2025-12-04T09:13:45.2944107Z ___________________ TestTraversalCUDA.test_fsdp_modules_cuda ___________________ 2025-12-04T09:13:45.2944579Z Traceback (most recent call last): 2025-12-04T09:13:45.2945270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:45.2945985Z self._join_processes(fn) 2025-12-04T09:13:45.2946703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:45.2947533Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:45.2948584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:45.2949447Z raise RuntimeError(error) 2025-12-04T09:13:45.2949900Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:13:45.2950385Z Traceback (most recent call last): 2025-12-04T09:13:45.2951172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.2951968Z getattr(self, test_name)() 2025-12-04T09:13:45.2952710Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.2953486Z fn() 2025-12-04T09:13:45.2954142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2954902Z method(*args, **kwargs) 2025-12-04T09:13:45.2955606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2956370Z method(*args, **kwargs) 2025-12-04T09:13:45.2957088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.2957843Z with policy(): 2025-12-04T09:13:45.2958525Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.2959291Z raise RuntimeError(msg) 2025-12-04T09:13:45.2960641Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.2961752Z 2025-12-04T09:13:45.2961969Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.2962796Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.2963447Z 2025-12-04T09:13:45.2963701Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.2964078Z 2025-12-04T09:13:45.2964087Z 2025-12-04T09:13:45.2964494Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:45.2965058Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:45.2966165Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-8583a5ccabf54360.xml - 2025-12-04T09:13:45.2967206Z =========================== short test summary info ============================ 2025-12-04T09:13:45.2968344Z FAILED [6.7291s] distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:13:45.2969269Z Traceback (most recent call last): 2025-12-04T09:13:45.2970003Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.2970759Z getattr(self, test_name)() 2025-12-04T09:13:45.2971537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.2972257Z fn() 2025-12-04T09:13:45.2972867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2973586Z method(*args, **kwargs) 2025-12-04T09:13:45.2974264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.2974967Z method(*args, **kwargs) 2025-12-04T09:13:45.2975636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.2976349Z with policy(): 2025-12-04T09:13:45.2976981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.2977708Z raise RuntimeError(msg) 2025-12-04T09:13:45.2978901Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.2980024Z 2025-12-04T09:13:45.2980240Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.2981068Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.2981711Z 2025-12-04T09:13:45.2981961Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.2982521Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:45.2982976Z ============================== 1 failed in 6.95s =============================== 2025-12-04T09:13:45.2983342Z Got exit code 1 2025-12-04T09:13:45.2983598Z Retrying single test... 2025-12-04T09:13:45.2984429Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-58c9173ffcd43d6d.xml 2025-12-04T09:13:45.2985450Z ============================= test session starts ============================== 2025-12-04T09:13:45.2986027Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:45.2986558Z cachedir: .pytest_cache 2025-12-04T09:13:45.2987186Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:45.2988150Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:45.2988505Z configfile: pytest.ini 2025-12-04T09:13:45.2989231Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:45.2990023Z collecting ... collected 1 item 2025-12-04T09:13:45.2990938Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda 2025-12-04T09:13:45.2991809Z Running 1 items in this shard 2025-12-04T09:13:45.2992016Z 2025-12-04T09:13:45.2992962Z distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda I1204 09:13:33.944000 23154 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 23206 2025-12-04T09:13:45.2994507Z I1204 09:13:33.945000 23154 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 23207 2025-12-04T09:13:45.2995597Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:45.2996731Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:45.2998423Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.3000241Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:45.3001776Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.3003219Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:45.3004637Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.3006139Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:45.3007729Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.3009134Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:45.3010554Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.3011926Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:45.3013317Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.3014748Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:45.3016643Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 0. CUDA driver allocated memory was 453967872 and is now 456065024. 2025-12-04T09:13:45.3018428Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:45.3019467Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.3021347Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.3022753Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:45.3024230Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.3025809Z [rank0]:E1204 09:13:38.931000 23206 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:13:45.3026953Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:13:45.3028159Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:13:45.3029958Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.3031618Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:13:45.3033261Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.3034800Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:13:45.3036314Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.3037900Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:45.3039585Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.3041004Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:13:45.3042423Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.3043795Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:13:45.3045173Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.3046598Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:13:45.3048500Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.3050278Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:45.3051434Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.3053018Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.3054343Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:13:45.3055431Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.3056683Z [rank1]:E1204 09:13:38.931000 23207 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:13:45.3057377Z dist init r=1, world=2 2025-12-04T09:13:45.3057642Z dist init r=0, world=2 2025-12-04T09:13:45.3058903Z [rank0]:[W1204 09:13:39.235196392 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:13:45.3060144Z FAILED [6.5958s] [100%] 2025-12-04T09:13:45.3060303Z 2025-12-04T09:13:45.3060439Z =================================== FAILURES =================================== 2025-12-04T09:13:45.3060935Z ___________________ TestTraversalCUDA.test_fsdp_modules_cuda ___________________ 2025-12-04T09:13:45.3061403Z Traceback (most recent call last): 2025-12-04T09:13:45.3062105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:13:45.3062804Z self._join_processes(fn) 2025-12-04T09:13:45.3063715Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:13:45.3064546Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:13:45.3065365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:13:45.3066181Z raise RuntimeError(error) 2025-12-04T09:13:45.3066603Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:13:45.3067068Z Traceback (most recent call last): 2025-12-04T09:13:45.3068037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.3068850Z getattr(self, test_name)() 2025-12-04T09:13:45.3069609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.3070386Z fn() 2025-12-04T09:13:45.3071029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.3071800Z method(*args, **kwargs) 2025-12-04T09:13:45.3072518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.3073265Z method(*args, **kwargs) 2025-12-04T09:13:45.3073977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.3074728Z with policy(): 2025-12-04T09:13:45.3075411Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.3076162Z raise RuntimeError(msg) 2025-12-04T09:13:45.3077430Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.3078606Z 2025-12-04T09:13:45.3078906Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.3079906Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.3080629Z 2025-12-04T09:13:45.3080868Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.3081242Z 2025-12-04T09:13:45.3081247Z 2025-12-04T09:13:45.3081449Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:13:45.3082010Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:13:45.3083138Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-58c9173ffcd43d6d.xml - 2025-12-04T09:13:45.3084163Z =========================== short test summary info ============================ 2025-12-04T09:13:45.3085153Z FAILED [6.5958s] distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:13:45.3086021Z Traceback (most recent call last): 2025-12-04T09:13:45.3086725Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:13:45.3087426Z getattr(self, test_name)() 2025-12-04T09:13:45.3088099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:13:45.3088782Z fn() 2025-12-04T09:13:45.3089347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.3090022Z method(*args, **kwargs) 2025-12-04T09:13:45.3090655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:13:45.3091327Z method(*args, **kwargs) 2025-12-04T09:13:45.3091965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:13:45.3092627Z with policy(): 2025-12-04T09:13:45.3093243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:13:45.3093930Z raise RuntimeError(msg) 2025-12-04T09:13:45.3095061Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestTraversalCUDA.test_fsdp_modules_cuda! Caching allocator allocated memory was 512 and is now reported as 2560 on device 1. CUDA driver allocated memory was 344915968 and is now 347013120. 2025-12-04T09:13:45.3096101Z 2025-12-04T09:13:45.3096294Z To execute this test, run the following from the base repo dir: 2025-12-04T09:13:45.3097082Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_traversal.py TestTraversalCUDA.test_fsdp_modules_cuda 2025-12-04T09:13:45.3097689Z 2025-12-04T09:13:45.3097929Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:13:45.3098454Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:13:45.3098870Z ============================== 1 failed in 6.81s =============================== 2025-12-04T09:13:45.3099229Z Got exit code 1 2025-12-04T09:13:45.3099783Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda 2025-12-04T09:13:45.3100667Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:13:45.3101763Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-3a68936c644e7b41.xml 2025-12-04T09:13:45.3102638Z ============================= test session starts ============================== 2025-12-04T09:13:45.3103225Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:13:45.3103808Z cachedir: .pytest_cache 2025-12-04T09:13:45.3104439Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:13:45.3105132Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:13:45.3105451Z configfile: pytest.ini 2025-12-04T09:13:45.3106080Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:13:45.3106867Z collecting ... collected 1 item / 1 deselected / 0 selected 2025-12-04T09:13:45.3107369Z stepcurrent: skipping 1 already run items. 2025-12-04T09:13:45.3107885Z Running 0 items in this shard 2025-12-04T09:13:45.3108107Z 2025-12-04T09:13:45.3108973Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-3a68936c644e7b41.xml - 2025-12-04T09:13:45.3110209Z ============================ 1 deselected in 0.01s ============================= 2025-12-04T09:13:45.3111096Z The following tests failed consistently: ['test/distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda'] 2025-12-04T09:13:45.3111792Z 2025-12-04T09:13:45.3112454Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal 1/1 (test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_e348e090749a4a32_.log) 2025-12-04T09:13:45.3113260Z 2025-12-04T09:13:45.3113667Z Finished distributed/fsdp/test_fsdp_traversal 1/1 ... [2025-12-04 09:13:45.278818][1256.216451831], took 0.62min 2025-12-04T09:13:45.3115142Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-551e66d027b33cdd.xml 2025-12-04T09:13:45.3615264Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-8583a5ccabf54360.xml 2025-12-04T09:13:45.3932072Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-58c9173ffcd43d6d.xml 2025-12-04T09:13:45.4293395Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-3a68936c644e7b41.xml 2025-12-04T09:13:45.6050411Z Uploading logs for 57116084912 to S3 2025-12-04T09:13:45.6383188Z Uploading artifacts took 0.18 seconds 2025-12-04T09:13:45.6384151Z distributed/fsdp/test_fsdp_traversal 1/1 failed! 2025-12-04T09:13:45.6386627Z Running distributed/fsdp/test_fsdp_checkpoint 1/1 ... [2025-12-04 09:13:45.638366][1256.576002608] 2025-12-04T09:13:45.6387347Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:13:45.6388896Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:13:45.638667] 2025-12-04T09:16:18.2854478Z 2025-12-04T09:16:18.2855561Z PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint 1/1 (test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_b53e20b9533041ba_.log) 2025-12-04T09:16:18.2857174Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-88a9f26e5fb62840.xml 2025-12-04T09:16:18.2858283Z ============================= test session starts ============================== 2025-12-04T09:16:18.2859024Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:16:18.2859686Z cachedir: .pytest_cache 2025-12-04T09:16:18.2860449Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:16:18.2861653Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:16:18.2862081Z configfile: pytest.ini 2025-12-04T09:16:18.2862868Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:16:18.2864779Z collecting ... /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:292: PytestCollectionWarning: cannot collect test class 'TestModel' because it has a __init__ constructor (from: test/distributed/fsdp/test_fsdp_checkpoint.py) 2025-12-04T09:16:18.2866323Z class TestModel(nn.Module): 2025-12-04T09:16:18.2866696Z collected 17 items 2025-12-04T09:16:18.2867039Z stepcurrent: Cannot find last run test, not skipping 2025-12-04T09:16:18.2882055Z Running 17 items in this shard: test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.2896205Z 2025-12-04T09:16:18.2897629Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_False I1204 09:13:49.054000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 23449 2025-12-04T09:16:18.2899545Z I1204 09:13:49.055000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 23450 2025-12-04T09:16:18.2900639Z I1204 09:13:49.056000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 23451 2025-12-04T09:16:18.2901807Z I1204 09:13:49.057000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 23452 2025-12-04T09:16:18.2903410Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2904624Z return func(*args, **kwargs) 2025-12-04T09:16:18.2904994Z dist init r=0, world=4 2025-12-04T09:16:18.2905270Z dist init r=3, world=4 2025-12-04T09:16:18.2905545Z dist init r=2, world=4 2025-12-04T09:16:18.2905803Z dist init r=1, world=4 2025-12-04T09:16:18.2906087Z PASSED [6.8717s] [ 5%] 2025-12-04T09:16:18.2907771Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_True I1204 09:13:55.883000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 23709 2025-12-04T09:16:18.2909737Z I1204 09:13:55.884000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 23710 2025-12-04T09:16:18.2910861Z I1204 09:13:55.885000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 23711 2025-12-04T09:16:18.2911994Z I1204 09:13:55.886000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 23712 2025-12-04T09:16:18.2913724Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2914936Z return func(*args, **kwargs) 2025-12-04T09:16:18.2915233Z dist init r=0, world=4 2025-12-04T09:16:18.2915508Z dist init r=3, world=4 2025-12-04T09:16:18.2915781Z dist init r=2, world=4 2025-12-04T09:16:18.2916044Z dist init r=1, world=4 2025-12-04T09:16:18.2916322Z PASSED [6.8188s] [ 11%] 2025-12-04T09:16:18.2917729Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_False I1204 09:14:02.704000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 23969 2025-12-04T09:16:18.2919738Z I1204 09:14:02.705000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 23970 2025-12-04T09:16:18.2920800Z I1204 09:14:02.706000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 23971 2025-12-04T09:16:18.2921867Z I1204 09:14:02.706000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 23972 2025-12-04T09:16:18.2923416Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2924968Z return func(*args, **kwargs) 2025-12-04T09:16:18.2925275Z dist init r=2, world=4 2025-12-04T09:16:18.2925587Z dist init r=1, world=4 2025-12-04T09:16:18.2925859Z dist init r=0, world=4 2025-12-04T09:16:18.2926139Z dist init r=3, world=4 2025-12-04T09:16:18.2926428Z PASSED [6.7187s] [ 17%] 2025-12-04T09:16:18.2927850Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_True I1204 09:14:09.424000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 24229 2025-12-04T09:16:18.2929736Z I1204 09:14:09.425000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 24230 2025-12-04T09:16:18.2930873Z I1204 09:14:09.426000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 24231 2025-12-04T09:16:18.2932136Z I1204 09:14:09.427000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 24232 2025-12-04T09:16:18.2933763Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2935012Z return func(*args, **kwargs) 2025-12-04T09:16:18.2935333Z dist init r=0, world=4 2025-12-04T09:16:18.2935621Z dist init r=2, world=4 2025-12-04T09:16:18.2935892Z dist init r=1, world=4 2025-12-04T09:16:18.2936304Z dist init r=3, world=4 2025-12-04T09:16:18.2936682Z PASSED [6.8188s] [ 23%] 2025-12-04T09:16:18.2938027Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_False I1204 09:14:16.245000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 24489 2025-12-04T09:16:18.2939886Z I1204 09:14:16.246000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 24490 2025-12-04T09:16:18.2941140Z I1204 09:14:16.247000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 24491 2025-12-04T09:16:18.2942243Z I1204 09:14:16.247000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 24492 2025-12-04T09:16:18.2943827Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2945106Z return func(*args, **kwargs) 2025-12-04T09:16:18.2945420Z dist init r=1, world=4 2025-12-04T09:16:18.2945696Z dist init r=2, world=4 2025-12-04T09:16:18.2945955Z dist init r=0, world=4 2025-12-04T09:16:18.2946231Z dist init r=3, world=4 2025-12-04T09:16:18.2946502Z PASSED [6.7185s] [ 29%] 2025-12-04T09:16:18.2948142Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_True I1204 09:14:22.965000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 24749 2025-12-04T09:16:18.2950018Z I1204 09:14:22.966000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 24750 2025-12-04T09:16:18.2951154Z I1204 09:14:22.967000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 24751 2025-12-04T09:16:18.2952287Z I1204 09:14:22.968000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 24752 2025-12-04T09:16:18.2953945Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2955186Z return func(*args, **kwargs) 2025-12-04T09:16:18.2955510Z dist init r=1, world=4 2025-12-04T09:16:18.2955794Z dist init r=0, world=4 2025-12-04T09:16:18.2956065Z dist init r=3, world=4 2025-12-04T09:16:18.2956343Z dist init r=2, world=4 2025-12-04T09:16:18.2956624Z PASSED [6.8193s] [ 35%] 2025-12-04T09:16:18.2958063Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_False I1204 09:14:29.786000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 25009 2025-12-04T09:16:18.2960193Z I1204 09:14:29.787000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 25010 2025-12-04T09:16:18.2961261Z I1204 09:14:29.788000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 25011 2025-12-04T09:16:18.2962425Z I1204 09:14:29.788000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 25012 2025-12-04T09:16:18.2963968Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2965125Z return func(*args, **kwargs) 2025-12-04T09:16:18.2965429Z dist init r=2, world=4 2025-12-04T09:16:18.2965698Z dist init r=1, world=4 2025-12-04T09:16:18.2965951Z dist init r=3, world=4 2025-12-04T09:16:18.2966220Z dist init r=0, world=4 2025-12-04T09:16:18.2966487Z PASSED [6.7187s] [ 41%] 2025-12-04T09:16:18.2967844Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_True I1204 09:14:36.506000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 25269 2025-12-04T09:16:18.2969780Z I1204 09:14:36.507000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 25270 2025-12-04T09:16:18.2970786Z I1204 09:14:36.508000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 25271 2025-12-04T09:16:18.2971795Z I1204 09:14:36.509000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 25272 2025-12-04T09:16:18.2973258Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2974354Z return func(*args, **kwargs) 2025-12-04T09:16:18.2974644Z dist init r=1, world=4 2025-12-04T09:16:18.2974900Z dist init r=3, world=4 2025-12-04T09:16:18.2975155Z dist init r=2, world=4 2025-12-04T09:16:18.2975394Z dist init r=0, world=4 2025-12-04T09:16:18.2975650Z PASSED [6.7185s] [ 47%] 2025-12-04T09:16:18.2976937Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_False I1204 09:14:43.226000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 25529 2025-12-04T09:16:18.2978601Z I1204 09:14:43.227000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 25530 2025-12-04T09:16:18.2979609Z I1204 09:14:43.228000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 25531 2025-12-04T09:16:18.2980619Z I1204 09:14:43.229000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 25532 2025-12-04T09:16:18.2982063Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2983177Z return func(*args, **kwargs) 2025-12-04T09:16:18.2983456Z dist init r=3, world=4 2025-12-04T09:16:18.2983713Z dist init r=1, world=4 2025-12-04T09:16:18.2983966Z dist init r=0, world=4 2025-12-04T09:16:18.2984203Z dist init r=2, world=4 2025-12-04T09:16:18.2984455Z PASSED [6.8187s] [ 52%] 2025-12-04T09:16:18.2985730Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_True I1204 09:14:50.047000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 25789 2025-12-04T09:16:18.2987468Z I1204 09:14:50.048000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 25790 2025-12-04T09:16:18.2988753Z I1204 09:14:50.048000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 25791 2025-12-04T09:16:18.2989951Z I1204 09:14:50.049000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 25792 2025-12-04T09:16:18.2991595Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.2992842Z return func(*args, **kwargs) 2025-12-04T09:16:18.2993147Z dist init r=0, world=4 2025-12-04T09:16:18.2993427Z dist init r=1, world=4 2025-12-04T09:16:18.2993707Z dist init r=3, world=4 2025-12-04T09:16:18.2993972Z dist init r=2, world=4 2025-12-04T09:16:18.2994254Z PASSED [6.8191s] [ 58%] 2025-12-04T09:16:18.2995698Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_False I1204 09:14:56.867000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 26049 2025-12-04T09:16:18.2997832Z I1204 09:14:56.868000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 26050 2025-12-04T09:16:18.2998955Z I1204 09:14:56.869000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 26051 2025-12-04T09:16:18.3000521Z I1204 09:14:56.870000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 26052 2025-12-04T09:16:18.3002061Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.3003239Z return func(*args, **kwargs) 2025-12-04T09:16:18.3003530Z dist init r=0, world=4 2025-12-04T09:16:18.3003798Z dist init r=2, world=4 2025-12-04T09:16:18.3004060Z dist init r=1, world=4 2025-12-04T09:16:18.3004310Z dist init r=3, world=4 2025-12-04T09:16:18.3004574Z PASSED [6.7183s] [ 64%] 2025-12-04T09:16:18.3008102Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_True I1204 09:15:03.587000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 26309 2025-12-04T09:16:18.3009949Z I1204 09:15:03.588000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 26310 2025-12-04T09:16:18.3011037Z I1204 09:15:03.589000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 26311 2025-12-04T09:16:18.3012141Z I1204 09:15:03.590000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 26312 2025-12-04T09:16:18.3013738Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.3014959Z return func(*args, **kwargs) 2025-12-04T09:16:18.3015265Z dist init r=0, world=4 2025-12-04T09:16:18.3015544Z dist init r=1, world=4 2025-12-04T09:16:18.3015820Z dist init r=2, world=4 2025-12-04T09:16:18.3016084Z dist init r=3, world=4 2025-12-04T09:16:18.3016359Z PASSED [6.7186s] [ 70%] 2025-12-04T09:16:18.3017767Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_False I1204 09:15:10.307000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 26569 2025-12-04T09:16:18.3019596Z I1204 09:15:10.308000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 26570 2025-12-04T09:16:18.3020688Z I1204 09:15:10.309000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 26571 2025-12-04T09:16:18.3021789Z I1204 09:15:10.310000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 26572 2025-12-04T09:16:18.3023483Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.3025122Z return func(*args, **kwargs) 2025-12-04T09:16:18.3025431Z dist init r=1, world=4 2025-12-04T09:16:18.3025717Z dist init r=2, world=4 2025-12-04T09:16:18.3026000Z dist init r=3, world=4 2025-12-04T09:16:18.3026266Z dist init r=0, world=4 2025-12-04T09:16:18.3026550Z PASSED [6.8191s] [ 76%] 2025-12-04T09:16:18.3028100Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_True I1204 09:15:17.128000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 26829 2025-12-04T09:16:18.3029989Z I1204 09:15:17.129000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 26830 2025-12-04T09:16:18.3031238Z I1204 09:15:17.130000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 26831 2025-12-04T09:16:18.3032366Z I1204 09:15:17.131000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 26832 2025-12-04T09:16:18.3034006Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.3035260Z return func(*args, **kwargs) 2025-12-04T09:16:18.3035567Z dist init r=1, world=4 2025-12-04T09:16:18.3035848Z dist init r=3, world=4 2025-12-04T09:16:18.3036128Z dist init r=2, world=4 2025-12-04T09:16:18.3036396Z dist init r=0, world=4 2025-12-04T09:16:18.3036675Z PASSED [6.8191s] [ 82%] 2025-12-04T09:16:18.3038117Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_False I1204 09:15:23.949000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 27089 2025-12-04T09:16:18.3040338Z I1204 09:15:23.950000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 27090 2025-12-04T09:16:18.3041422Z I1204 09:15:23.951000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 27091 2025-12-04T09:16:18.3042518Z I1204 09:15:23.951000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 27092 2025-12-04T09:16:18.3044109Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.3045316Z return func(*args, **kwargs) 2025-12-04T09:16:18.3045620Z dist init r=3, world=4 2025-12-04T09:16:18.3045899Z dist init r=1, world=4 2025-12-04T09:16:18.3046169Z dist init r=0, world=4 2025-12-04T09:16:18.3046437Z dist init r=2, world=4 2025-12-04T09:16:18.3046696Z PASSED [6.8196s] [ 88%] 2025-12-04T09:16:18.3048235Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_True I1204 09:15:30.770000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 27349 2025-12-04T09:16:18.3050065Z I1204 09:15:30.771000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 27350 2025-12-04T09:16:18.3051161Z I1204 09:15:30.772000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 27351 2025-12-04T09:16:18.3052240Z I1204 09:15:30.773000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 27352 2025-12-04T09:16:18.3053906Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning. 2025-12-04T09:16:18.3055223Z return func(*args, **kwargs) 2025-12-04T09:16:18.3055532Z dist init r=2, world=4 2025-12-04T09:16:18.3055791Z dist init r=0, world=4 2025-12-04T09:16:18.3056067Z dist init r=3, world=4 2025-12-04T09:16:18.3056336Z dist init r=1, world=4 2025-12-04T09:16:18.3056590Z PASSED [6.8193s] [ 94%] 2025-12-04T09:16:18.3057974Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda I1204 09:15:37.592000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 27609 2025-12-04T09:16:18.3059561Z I1204 09:15:37.593000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 27610 2025-12-04T09:16:18.3060633Z I1204 09:15:37.594000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 27611 2025-12-04T09:16:18.3061626Z I1204 09:15:37.595000 23397 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 27612 2025-12-04T09:16:18.3063057Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3064242Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3065420Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3066772Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3069086Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3071116Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3073122Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3075130Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3076377Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3077717Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3079905Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3081810Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3082985Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3084297Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3085550Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3086801Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3088046Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3089271Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3090617Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3093347Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3094553Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3095734Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3096936Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3098126Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3099323Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3100514Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3101705Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3102893Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3104093Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3105284Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3110148Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3115152Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3120432Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3125497Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3130633Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3135663Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3137284Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3138472Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3140335Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3142123Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3143229Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3144413Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3145596Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3146788Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3148255Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3149599Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3154584Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3159700Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3160791Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3161807Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3163322Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3164780Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3166250Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3167621Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3168958Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3170379Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3171785Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3173201Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3174622Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3176002Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3177388Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3178805Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3180974Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 1. CUDA driver allocated memory was 604962816 and is now 638517248. 2025-12-04T09:16:18.3182984Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3184026Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3185839Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3187460Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3188852Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3190336Z [rank1]:E1204 09:15:44.394000 27610 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:16:18.3191485Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3192610Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3194276Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3195930Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3197579Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3199127Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3200762Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3202186Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3203606Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3205029Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3206448Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3207822Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3209207Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3210643Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3212821Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 2. CUDA driver allocated memory was 604962816 and is now 638517248. 2025-12-04T09:16:18.3214840Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3215875Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3217688Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3219307Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3220404Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3221653Z [rank2]:E1204 09:15:44.395000 27611 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:16:18.3222658Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3223819Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3225668Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3227415Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3229067Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3230596Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3232108Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3233705Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3235310Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3236907Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3238484Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3240165Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3241715Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3243151Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3245294Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 3. CUDA driver allocated memory was 581894144 and is now 638517248. 2025-12-04T09:16:18.3247285Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3248329Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3250232Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3251783Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3252871Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3254111Z [rank3]:E1204 09:15:44.397000 27612 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:16:18.3255129Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3256144Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3257641Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3259103Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3260574Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3261941Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3263292Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3264716Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3266117Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3267603Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3269369Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3270993Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3272559Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3274150Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3276552Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 0. CUDA driver allocated memory was 714014720 and is now 747569152. 2025-12-04T09:16:18.3278805Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3280246Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3282061Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3283605Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3284700Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3285956Z [rank0]:E1204 09:15:44.409000 27609 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:16:18.3286675Z dist init r=1, world=4 2025-12-04T09:16:18.3286929Z dist init r=3, world=4 2025-12-04T09:16:18.3287187Z dist init r=2, world=4 2025-12-04T09:16:18.3287444Z dist init r=0, world=4 2025-12-04T09:16:18.3288633Z [rank0]:[W1204 09:15:44.809195486 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:16:18.3289873Z FAILED [8.2227s] [100%] 2025-12-04T09:16:18.3290047Z 2025-12-04T09:16:18.3290186Z =================================== FAILURES =================================== 2025-12-04T09:16:18.3290787Z _ TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda _ 2025-12-04T09:16:18.3291350Z Traceback (most recent call last): 2025-12-04T09:16:18.3292058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:16:18.3292785Z self._join_processes(fn) 2025-12-04T09:16:18.3293500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:16:18.3294269Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:16:18.3295062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:16:18.3295833Z raise RuntimeError(error) 2025-12-04T09:16:18.3296224Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:16:18.3296666Z Traceback (most recent call last): 2025-12-04T09:16:18.3297361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3298076Z getattr(self, test_name)() 2025-12-04T09:16:18.3298795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3299489Z fn() 2025-12-04T09:16:18.3300069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3300746Z method(*args, **kwargs) 2025-12-04T09:16:18.3301368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3302043Z method(*args, **kwargs) 2025-12-04T09:16:18.3302686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3303342Z with policy(): 2025-12-04T09:16:18.3303954Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3304635Z raise RuntimeError(msg) 2025-12-04T09:16:18.3305982Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 1. CUDA driver allocated memory was 604962816 and is now 638517248. 2025-12-04T09:16:18.3307400Z 2025-12-04T09:16:18.3307597Z To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3308907Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3309849Z 2025-12-04T09:16:18.3310115Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3310517Z 2025-12-04T09:16:18.3310522Z 2025-12-04T09:16:18.3310764Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:16:18.3311380Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:16:18.3312664Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-88a9f26e5fb62840.xml - 2025-12-04T09:16:18.3313844Z =========================== short test summary info ============================ 2025-12-04T09:16:18.3315136Z FAILED [8.2227s] distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:16:18.3316348Z Traceback (most recent call last): 2025-12-04T09:16:18.3317125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3317930Z getattr(self, test_name)() 2025-12-04T09:16:18.3318685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3319441Z fn() 2025-12-04T09:16:18.3320296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3321015Z method(*args, **kwargs) 2025-12-04T09:16:18.3321690Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3322393Z method(*args, **kwargs) 2025-12-04T09:16:18.3323071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3324778Z with policy(): 2025-12-04T09:16:18.3325459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3326230Z raise RuntimeError(msg) 2025-12-04T09:16:18.3327880Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 1. CUDA driver allocated memory was 604962816 and is now 638517248. 2025-12-04T09:16:18.3329318Z 2025-12-04T09:16:18.3329551Z To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3330702Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3331625Z 2025-12-04T09:16:18.3331892Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3332487Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:16:18.3333011Z =================== 1 failed, 16 passed in 116.83s (0:01:56) =================== 2025-12-04T09:16:18.3333433Z Got exit code 1 2025-12-04T09:16:18.3333706Z Retrying single test... 2025-12-04T09:16:18.3334592Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-52aebc2b2ad7446f.xml 2025-12-04T09:16:18.3335688Z ============================= test session starts ============================== 2025-12-04T09:16:18.3336343Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:16:18.3337048Z cachedir: .pytest_cache 2025-12-04T09:16:18.3337861Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:16:18.3338582Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:16:18.3338914Z configfile: pytest.ini 2025-12-04T09:16:18.3339603Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:16:18.3341339Z collecting ... /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:292: PytestCollectionWarning: cannot collect test class 'TestModel' because it has a __init__ constructor (from: test/distributed/fsdp/test_fsdp_checkpoint.py) 2025-12-04T09:16:18.3342667Z class TestModel(nn.Module): 2025-12-04T09:16:18.3343021Z collected 17 items / 16 deselected / 1 selected 2025-12-04T09:16:18.3344109Z stepcurrent: skipping 16 already run items. Running only test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3345156Z Running 1 items in this shard 2025-12-04T09:16:18.3345355Z 2025-12-04T09:16:18.3346442Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda I1204 09:15:50.954000 27894 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 27946 2025-12-04T09:16:18.3348414Z I1204 09:15:50.955000 27894 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 27947 2025-12-04T09:16:18.3349554Z I1204 09:15:50.956000 27894 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 27948 2025-12-04T09:16:18.3350698Z I1204 09:15:50.957000 27894 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 27949 2025-12-04T09:16:18.3352297Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3353619Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3355729Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3357759Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3359083Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3360424Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3362277Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3364067Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3365182Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3366414Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3368272Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3370062Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3371175Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3372360Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3373553Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3374715Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3375891Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3377085Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3378286Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3379447Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3380632Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3381825Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3383028Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3384208Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3385405Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3386595Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3388129Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3389477Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3390812Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3392151Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3397062Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3402096Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3406536Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3410951Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3415399Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3419809Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3421204Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3422387Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3424795Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3426810Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3428130Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3429569Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3430896Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3432233Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3433572Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3434908Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3439857Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3444286Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3445180Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3446189Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3447678Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3449140Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3450595Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3452131Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3453624Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3455132Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3456620Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3458124Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3459635Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3461164Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3462633Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3464127Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3466391Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 2. CUDA driver allocated memory was 607059968 and is now 638517248. 2025-12-04T09:16:18.3468873Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3470042Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3472092Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3473834Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3475066Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3476489Z [rank2]:E1204 09:15:57.755000 27948 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:16:18.3477639Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3478759Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3480584Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3482053Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3483567Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3484939Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3486267Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3487687Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3489110Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3490524Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3491989Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3493359Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3494749Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3496168Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3498298Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 1. CUDA driver allocated memory was 604962816 and is now 638517248. 2025-12-04T09:16:18.3500307Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3501333Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3503148Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3504712Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3505812Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3507071Z [rank1]:E1204 09:15:57.755000 27947 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:16:18.3508408Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3509537Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3511227Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3512949Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3514595Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3516116Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3517625Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3519231Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3520866Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3522414Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3524279Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3525841Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3527412Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3529044Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3531445Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 3. CUDA driver allocated memory was 581894144 and is now 638517248. 2025-12-04T09:16:18.3533707Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3534882Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3537083Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3538645Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3539730Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3540986Z [rank3]:E1204 09:15:57.756000 27949 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:16:18.3542003Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3543013Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3544617Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3546082Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3547775Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3549315Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3550830Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3552522Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3554114Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3567581Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3569030Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3570438Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3571870Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3573297Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3575424Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 0. CUDA driver allocated memory was 714014720 and is now 747569152. 2025-12-04T09:16:18.3577429Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3578458Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3580282Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3581831Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3582925Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3584173Z [rank0]:E1204 09:15:57.769000 27946 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:16:18.3584887Z dist init r=1, world=4 2025-12-04T09:16:18.3585157Z dist init r=2, world=4 2025-12-04T09:16:18.3585414Z dist init r=3, world=4 2025-12-04T09:16:18.3585770Z dist init r=0, world=4 2025-12-04T09:16:18.3586973Z [rank0]:[W1204 09:15:58.139446754 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:16:18.3588588Z FAILED [8.5039s] [100%] 2025-12-04T09:16:18.3588770Z 2025-12-04T09:16:18.3588936Z =================================== FAILURES =================================== 2025-12-04T09:16:18.3589598Z _ TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda _ 2025-12-04T09:16:18.3590247Z Traceback (most recent call last): 2025-12-04T09:16:18.3591046Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:16:18.3591838Z self._join_processes(fn) 2025-12-04T09:16:18.3592733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:16:18.3593612Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:16:18.3594501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:16:18.3595358Z raise RuntimeError(error) 2025-12-04T09:16:18.3595813Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:16:18.3596316Z Traceback (most recent call last): 2025-12-04T09:16:18.3597086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3597888Z getattr(self, test_name)() 2025-12-04T09:16:18.3598631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3599407Z fn() 2025-12-04T09:16:18.3600159Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3600837Z method(*args, **kwargs) 2025-12-04T09:16:18.3601460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3602134Z method(*args, **kwargs) 2025-12-04T09:16:18.3602767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3603429Z with policy(): 2025-12-04T09:16:18.3604039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3604722Z raise RuntimeError(msg) 2025-12-04T09:16:18.3606079Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 2. CUDA driver allocated memory was 607059968 and is now 638517248. 2025-12-04T09:16:18.3607352Z 2025-12-04T09:16:18.3607546Z To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3608561Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3609388Z 2025-12-04T09:16:18.3609625Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3609982Z 2025-12-04T09:16:18.3609987Z 2025-12-04T09:16:18.3610199Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:16:18.3610763Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:16:18.3611887Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-52aebc2b2ad7446f.xml - 2025-12-04T09:16:18.3613009Z =========================== short test summary info ============================ 2025-12-04T09:16:18.3614169Z FAILED [8.5039s] distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:16:18.3615247Z Traceback (most recent call last): 2025-12-04T09:16:18.3615942Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3616661Z getattr(self, test_name)() 2025-12-04T09:16:18.3617342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3618023Z fn() 2025-12-04T09:16:18.3618601Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3619358Z method(*args, **kwargs) 2025-12-04T09:16:18.3619996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3620658Z method(*args, **kwargs) 2025-12-04T09:16:18.3621295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3621968Z with policy(): 2025-12-04T09:16:18.3622565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3623248Z raise RuntimeError(msg) 2025-12-04T09:16:18.3625021Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 2. CUDA driver allocated memory was 607059968 and is now 638517248. 2025-12-04T09:16:18.3626476Z 2025-12-04T09:16:18.3626712Z To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3627946Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3628865Z 2025-12-04T09:16:18.3629135Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3629730Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:16:18.3630237Z ======================= 1 failed, 16 deselected in 8.54s ======================= 2025-12-04T09:16:18.3630647Z Got exit code 1 2025-12-04T09:16:18.3630921Z Retrying single test... 2025-12-04T09:16:18.3631813Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-c4e36de52e6d69c2.xml 2025-12-04T09:16:18.3632812Z ============================= test session starts ============================== 2025-12-04T09:16:18.3633472Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:16:18.3634075Z cachedir: .pytest_cache 2025-12-04T09:16:18.3634778Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:16:18.3635558Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:16:18.3635896Z configfile: pytest.ini 2025-12-04T09:16:18.3636621Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:16:18.3638464Z collecting ... /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:292: PytestCollectionWarning: cannot collect test class 'TestModel' because it has a __init__ constructor (from: test/distributed/fsdp/test_fsdp_checkpoint.py) 2025-12-04T09:16:18.3639919Z class TestModel(nn.Module): 2025-12-04T09:16:18.3640254Z collected 17 items / 16 deselected / 1 selected 2025-12-04T09:16:18.3641401Z stepcurrent: skipping 16 already run items. Running only test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3642391Z Running 1 items in this shard 2025-12-04T09:16:18.3642578Z 2025-12-04T09:16:18.3643614Z distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda I1204 09:16:04.344000 28231 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 28283 2025-12-04T09:16:18.3645209Z I1204 09:16:04.345000 28231 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 28284 2025-12-04T09:16:18.3646215Z I1204 09:16:04.346000 28231 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 28285 2025-12-04T09:16:18.3647225Z I1204 09:16:04.347000 28231 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 28286 2025-12-04T09:16:18.3648714Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3649886Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3651760Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3653553Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3654667Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3655855Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3657702Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3659484Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3660597Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3661785Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3663646Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3665442Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3666554Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3668002Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3669400Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3670739Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3672090Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3673424Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3674766Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3676072Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3677401Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3678848Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3680274Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3681462Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3682641Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3683812Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3684989Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3686180Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3687367Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3688570Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3689758Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:322: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3690921Z model.checkpoint1 = FSDP(module=model.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3692783Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:16:18.3694569Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:16:18.3698872Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3703310Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3708038Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3713097Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3718199Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3723304Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3725079Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:323: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3726416Z model.checkpoint2 = FSDP(module=model.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3727753Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:325: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3729098Z model_ac.checkpoint1 = FSDP(module=model_ac.checkpoint1, **fsdp_kwargs) 2025-12-04T09:16:18.3730436Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:326: FutureWarning: The `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead. 2025-12-04T09:16:18.3731780Z model_ac.checkpoint2 = FSDP(module=model_ac.checkpoint2, **fsdp_kwargs) 2025-12-04T09:16:18.3736962Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient. This may incur unnecessary synchronization and break CUDA graph capture if the AccumulateGrad node's stream is the default stream. This mismatch is caused by an AccumulateGrad node created prior to the current iteration being kept alive. This can happen if the autograd graph is still being kept alive by tensors such as the loss, or if you are using DDP, which will stash a reference to the node. To resolve the mismatch, delete all references to the autograd graph or ensure that DDP initialization is performed under the same stream as subsequent forwards. If the mismatch is intentional, you can use torch.autograd.graph.set_warn_on_accumulate_grad_stream_mismatch(False) to suppress this warning. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/input_buffer.cpp:240.) 2025-12-04T09:16:18.3741672Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:16:18.3742611Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3743760Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3745344Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3746884Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3748726Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3750267Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3751783Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3753390Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3754981Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3756584Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3758176Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3759845Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3761364Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3762775Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3764904Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 1. CUDA driver allocated memory was 607059968 and is now 638517248. 2025-12-04T09:16:18.3766972Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3768016Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3769841Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3771379Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3772471Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3773731Z [rank1]:E1204 09:16:11.142000 28284 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:16:18.3774806Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3775812Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3777287Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3778751Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3780212Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3781583Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3782909Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3784322Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3785749Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3787162Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3788957Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3790494Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3792059Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3793657Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3796122Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 2. CUDA driver allocated memory was 609157120 and is now 638517248. 2025-12-04T09:16:18.3798384Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3799542Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3801515Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3803064Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3804218Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3805478Z [rank2]:E1204 09:16:11.142000 28285 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:16:18.3806486Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3807495Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3808986Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3810447Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3811921Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3813286Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3814629Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3816039Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3817460Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3818881Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3820296Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3821909Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3823366Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3825254Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3827851Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 3. CUDA driver allocated memory was 607059968 and is now 638517248. 2025-12-04T09:16:18.3830126Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3831309Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3833346Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3835177Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3836414Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3837837Z [rank3]:E1204 09:16:11.142000 28286 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:16:18.3838975Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:16:18.3840241Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:16:18.3841740Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3843213Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:16:18.3844676Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3846026Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:16:18.3847373Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3848795Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3850214Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3851627Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:16:18.3853038Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3854412Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:16:18.3855859Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3857509Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:16:18.3859766Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 0. CUDA driver allocated memory was 714014720 and is now 747569152. 2025-12-04T09:16:18.3861887Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3863177Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3865223Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3866913Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:16:18.3868371Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3869771Z [rank0]:E1204 09:16:11.153000 28283 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:16:18.3870576Z dist init r=2, world=4 2025-12-04T09:16:18.3870867Z dist init r=1, world=4 2025-12-04T09:16:18.3871136Z dist init r=3, world=4 2025-12-04T09:16:18.3871415Z dist init r=0, world=4 2025-12-04T09:16:18.3872764Z [rank0]:[W1204 09:16:11.516428976 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:16:18.3874152Z FAILED [8.3954s] [100%] 2025-12-04T09:16:18.3874330Z 2025-12-04T09:16:18.3874481Z =================================== FAILURES =================================== 2025-12-04T09:16:18.3875156Z _ TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda _ 2025-12-04T09:16:18.3875796Z Traceback (most recent call last): 2025-12-04T09:16:18.3876589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:16:18.3877387Z self._join_processes(fn) 2025-12-04T09:16:18.3878186Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:16:18.3879068Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:16:18.3880022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:16:18.3880835Z raise RuntimeError(error) 2025-12-04T09:16:18.3881261Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:16:18.3881728Z Traceback (most recent call last): 2025-12-04T09:16:18.3882450Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3883202Z getattr(self, test_name)() 2025-12-04T09:16:18.3883919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3884725Z fn() 2025-12-04T09:16:18.3885309Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3886509Z method(*args, **kwargs) 2025-12-04T09:16:18.3887157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3887827Z method(*args, **kwargs) 2025-12-04T09:16:18.3888472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3889147Z with policy(): 2025-12-04T09:16:18.3889747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3890432Z raise RuntimeError(msg) 2025-12-04T09:16:18.3891773Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 2. CUDA driver allocated memory was 609157120 and is now 638517248. 2025-12-04T09:16:18.3893104Z 2025-12-04T09:16:18.3893320Z To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3894348Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3895167Z 2025-12-04T09:16:18.3895404Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3895774Z 2025-12-04T09:16:18.3895778Z 2025-12-04T09:16:18.3895979Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:16:18.3896545Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:16:18.3897687Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-c4e36de52e6d69c2.xml - 2025-12-04T09:16:18.3898737Z =========================== short test summary info ============================ 2025-12-04T09:16:18.3899891Z FAILED [8.3954s] distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:16:18.3900969Z Traceback (most recent call last): 2025-12-04T09:16:18.3901674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:16:18.3902383Z getattr(self, test_name)() 2025-12-04T09:16:18.3903056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:16:18.3903742Z fn() 2025-12-04T09:16:18.3904316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3904977Z method(*args, **kwargs) 2025-12-04T09:16:18.3905623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:16:18.3906295Z method(*args, **kwargs) 2025-12-04T09:16:18.3906917Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:16:18.3907845Z with policy(): 2025-12-04T09:16:18.3908539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:16:18.3909309Z raise RuntimeError(msg) 2025-12-04T09:16:18.3910814Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda! Caching allocator allocated memory was 512 and is now reported as 3236352 on device 2. CUDA driver allocated memory was 609157120 and is now 638517248. 2025-12-04T09:16:18.3912259Z 2025-12-04T09:16:18.3912476Z To execute this test, run the following from the base repo dir: 2025-12-04T09:16:18.3913681Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_checkpoint.py TestFSDPCheckpointSubmoduleCUDA.test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3914591Z 2025-12-04T09:16:18.3914852Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:16:18.3915418Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:16:18.3915903Z ======================= 1 failed, 16 deselected in 8.42s ======================= 2025-12-04T09:16:18.3916314Z Got exit code 1 2025-12-04T09:16:18.3917182Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda 2025-12-04T09:16:18.3918406Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:16:18.3919634Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-448a44a24c52f556.xml 2025-12-04T09:16:18.3920698Z ============================= test session starts ============================== 2025-12-04T09:16:18.3921262Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:16:18.3921775Z cachedir: .pytest_cache 2025-12-04T09:16:18.3922387Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:16:18.3923065Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:16:18.3923356Z configfile: pytest.ini 2025-12-04T09:16:18.3924343Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:16:18.3926182Z collecting ... /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_checkpoint.py:292: PytestCollectionWarning: cannot collect test class 'TestModel' because it has a __init__ constructor (from: test/distributed/fsdp/test_fsdp_checkpoint.py) 2025-12-04T09:16:18.3927593Z class TestModel(nn.Module): 2025-12-04T09:16:18.3927946Z collected 17 items / 17 deselected / 0 selected 2025-12-04T09:16:18.3928365Z stepcurrent: skipping 17 already run items. 2025-12-04T09:16:18.3928745Z Running 0 items in this shard 2025-12-04T09:16:18.3928949Z 2025-12-04T09:16:18.3929830Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-448a44a24c52f556.xml - 2025-12-04T09:16:18.3930983Z ============================ 17 deselected in 0.02s ============================ 2025-12-04T09:16:18.3932090Z The following tests failed consistently: ['test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda'] 2025-12-04T09:16:18.3933038Z 2025-12-04T09:16:18.3933714Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint 1/1 (test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_b53e20b9533041ba_.log) 2025-12-04T09:16:18.3934514Z 2025-12-04T09:16:18.3934935Z Finished distributed/fsdp/test_fsdp_checkpoint 1/1 ... [2025-12-04 09:16:18.285868][1409.223502095], took 2.54min 2025-12-04T09:16:18.3936514Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-88a9f26e5fb62840.xml 2025-12-04T09:16:18.3938282Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-52aebc2b2ad7446f.xml 2025-12-04T09:16:18.4012201Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-c4e36de52e6d69c2.xml 2025-12-04T09:16:18.4313818Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-448a44a24c52f556.xml 2025-12-04T09:16:18.7053727Z Uploading logs for 57116084912 to S3 2025-12-04T09:16:18.7510641Z Uploading artifacts took 0.29 seconds 2025-12-04T09:16:18.7511079Z distributed/fsdp/test_fsdp_checkpoint 1/1 failed! 2025-12-04T09:16:18.7520164Z Running distributed/fsdp/test_fsdp_comm 1/1 ... [2025-12-04 09:16:18.751471][1409.689107542] 2025-12-04T09:16:18.7520812Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:16:18.7522062Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:16:18.751782] 2025-12-04T09:22:55.3374110Z 2025-12-04T09:22:55.3375091Z PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm 1/1 (test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_8c711ca16ddf8d8f_.log) 2025-12-04T09:22:55.3377086Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a25290001c1b014e.xml 2025-12-04T09:22:55.3378059Z ============================= test session starts ============================== 2025-12-04T09:22:55.3378819Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.3379477Z cachedir: .pytest_cache 2025-12-04T09:22:55.3380241Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.3381059Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.3381469Z configfile: pytest.ini 2025-12-04T09:22:55.3382252Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.3383093Z collecting ... collected 10 items 2025-12-04T09:22:55.3383493Z stepcurrent: Cannot find last run test, not skipping 2025-12-04T09:22:55.3391764Z Running 10 items in this shard: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.3399951Z 2025-12-04T09:22:55.3401216Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda I1204 09:16:22.184000 28625 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 28677 2025-12-04T09:22:55.3403253Z I1204 09:16:22.185000 28625 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 28678 2025-12-04T09:22:55.3404541Z I1204 09:16:22.185000 28625 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 28679 2025-12-04T09:22:55.3405958Z I1204 09:16:22.186000 28625 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 28680 2025-12-04T09:22:55.3408062Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3409741Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3411304Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3412871Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3414443Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3416025Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3418066Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3420104Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3422110Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3424344Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3426396Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3428533Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3430092Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3431621Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3433583Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3435586Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3436354Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3437491Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3439363Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3441032Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3442690Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3444247Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3445760Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3447441Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3449049Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3450645Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3452240Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3453791Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3455343Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3456943Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3459386Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.3461776Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3462915Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3465021Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3466742Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3468212Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3469627Z [rank1]:E1204 09:16:29.047000 28678 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.3470854Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3471983Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3473663Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3475316Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3476960Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3478516Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3480172Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3481717Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3483269Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3484820Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3486363Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3487867Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3489387Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3490940Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3493321Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 581894144 and is now 634322944. 2025-12-04T09:22:55.3495557Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3496685Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3498711Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3500439Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3501635Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3503071Z [rank3]:E1204 09:16:29.051000 28680 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.3504170Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3505266Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3506901Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3508782Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3510419Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3512015Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3513519Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3515298Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3516973Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3518567Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3520240Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3521750Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3523260Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3525202Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3527633Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.3529969Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3531137Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3533212Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3536640Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3537961Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3539289Z [rank2]:E1204 09:16:29.051000 28679 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.3540039Z dist init r=2, world=4 2025-12-04T09:22:55.3540309Z dist init r=1, world=4 2025-12-04T09:22:55.3540900Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3541967Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3543548Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3545192Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3546727Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3548450Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3549962Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3551565Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3553167Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3554745Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3556340Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3557892Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3559459Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3561056Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3563207Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.3565260Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3566298Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3568205Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3569793Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3570872Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3572124Z [rank0]:E1204 09:16:29.166000 28677 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.3572839Z dist init r=3, world=4 2025-12-04T09:22:55.3573099Z dist init r=0, world=4 2025-12-04T09:22:55.3574289Z [rank0]:[W1204 09:16:29.485529419 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.3575570Z FAILED [9.1932s] [ 10%] 2025-12-04T09:22:55.3575746Z 2025-12-04T09:22:55.3575881Z =================================== FAILURES =================================== 2025-12-04T09:22:55.3576548Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda _ 2025-12-04T09:22:55.3577168Z Traceback (most recent call last): 2025-12-04T09:22:55.3577872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.3578586Z self._join_processes(fn) 2025-12-04T09:22:55.3579296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.3580060Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.3580855Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.3581623Z raise RuntimeError(error) 2025-12-04T09:22:55.3582015Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.3582457Z Traceback (most recent call last): 2025-12-04T09:22:55.3583153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3583862Z getattr(self, test_name)() 2025-12-04T09:22:55.3584522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3585210Z fn() 2025-12-04T09:22:55.3585785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3586452Z method(*args, **kwargs) 2025-12-04T09:22:55.3587091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3588034Z method(*args, **kwargs) 2025-12-04T09:22:55.3588748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3589489Z with policy(): 2025-12-04T09:22:55.3590170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3590936Z raise RuntimeError(msg) 2025-12-04T09:22:55.3592501Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.3593985Z 2025-12-04T09:22:55.3594276Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3595467Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3596444Z 2025-12-04T09:22:55.3596711Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3597113Z 2025-12-04T09:22:55.3597117Z 2025-12-04T09:22:55.3597359Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.3597979Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.3599180Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a25290001c1b014e.xml - 2025-12-04T09:22:55.3600460Z =========================== short test summary info ============================ 2025-12-04T09:22:55.3601691Z FAILED [9.1932s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.3602783Z Traceback (most recent call last): 2025-12-04T09:22:55.3603475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3604201Z getattr(self, test_name)() 2025-12-04T09:22:55.3604876Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3605549Z fn() 2025-12-04T09:22:55.3606130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3606813Z method(*args, **kwargs) 2025-12-04T09:22:55.3607456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3608123Z method(*args, **kwargs) 2025-12-04T09:22:55.3608763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3609441Z with policy(): 2025-12-04T09:22:55.3610040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3610732Z raise RuntimeError(msg) 2025-12-04T09:22:55.3612132Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.3613443Z 2025-12-04T09:22:55.3613655Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3614704Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3615567Z 2025-12-04T09:22:55.3615804Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3616402Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.3616855Z ============================== 1 failed in 9.22s =============================== 2025-12-04T09:22:55.3617262Z Got exit code 1 2025-12-04T09:22:55.3617510Z Retrying single test... 2025-12-04T09:22:55.3618247Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-cc80cfecc5abaf73.xml 2025-12-04T09:22:55.3619080Z ============================= test session starts ============================== 2025-12-04T09:22:55.3619729Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.3620268Z cachedir: .pytest_cache 2025-12-04T09:22:55.3620897Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.3621578Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.3621895Z configfile: pytest.ini 2025-12-04T09:22:55.3622537Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.3623325Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.3624803Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3625982Z Running 1 items in this shard 2025-12-04T09:22:55.3626192Z 2025-12-04T09:22:55.3627593Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda I1204 09:16:36.144000 28962 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 29014 2025-12-04T09:22:55.3629413Z I1204 09:16:36.145000 28962 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 29015 2025-12-04T09:22:55.3630542Z I1204 09:16:36.146000 28962 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 29016 2025-12-04T09:22:55.3631673Z I1204 09:16:36.147000 28962 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 29017 2025-12-04T09:22:55.3633552Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3635066Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3637028Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3639151Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3640643Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3642103Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3643537Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3644992Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3647266Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3649165Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3651144Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3653039Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3654489Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3655894Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3657741Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3659687Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3660404Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3661473Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3663053Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3664604Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3666153Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3667823Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3669340Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3670925Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3672527Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3674123Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3675730Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3677272Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3678833Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3680498Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3682724Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.3684777Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3685816Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3687649Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3689227Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3690375Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3691622Z [rank1]:E1204 09:16:42.888000 29015 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.3692625Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3693628Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3695118Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3696592Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3698054Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3699401Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3700736Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3702162Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3703586Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3705005Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3706410Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3708046Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3709604Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3711278Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3713725Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.3716007Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3717182Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3719266Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3721128Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3722221Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3723464Z [rank2]:E1204 09:16:42.910000 29016 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.3724875Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3726013Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3727712Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3729350Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3731002Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3732541Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3734052Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3735661Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3737271Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3738690Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3740110Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3741499Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3742984Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3744398Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3746575Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 491716608 and is now 634322944. 2025-12-04T09:22:55.3748979Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3750154Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3752319Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3754103Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3755325Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3756736Z [rank3]:E1204 09:16:42.915000 29017 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.3757539Z dist init r=1, world=4 2025-12-04T09:22:55.3757822Z dist init r=2, world=4 2025-12-04T09:22:55.3758113Z dist init r=3, world=4 2025-12-04T09:22:55.3758880Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3760263Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3761931Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3763520Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3765055Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3788555Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3790100Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3791729Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3793323Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3794921Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3796675Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3798241Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3799905Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3801454Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3803625Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.3805748Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3806802Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3808656Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3810240Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3811330Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3812585Z [rank0]:E1204 09:16:43.060000 29014 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.3813295Z dist init r=0, world=4 2025-12-04T09:22:55.3814496Z [rank0]:[W1204 09:16:43.402849560 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.3815714Z FAILED [8.5972s] [100%] 2025-12-04T09:22:55.3815892Z 2025-12-04T09:22:55.3816030Z =================================== FAILURES =================================== 2025-12-04T09:22:55.3816693Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda _ 2025-12-04T09:22:55.3817335Z Traceback (most recent call last): 2025-12-04T09:22:55.3818029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.3818743Z self._join_processes(fn) 2025-12-04T09:22:55.3819458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.3820222Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.3821007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.3821779Z raise RuntimeError(error) 2025-12-04T09:22:55.3822186Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.3822614Z Traceback (most recent call last): 2025-12-04T09:22:55.3823389Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3824497Z getattr(self, test_name)() 2025-12-04T09:22:55.3825262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3826038Z fn() 2025-12-04T09:22:55.3826676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3827519Z method(*args, **kwargs) 2025-12-04T09:22:55.3828237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3828997Z method(*args, **kwargs) 2025-12-04T09:22:55.3829695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3830443Z with policy(): 2025-12-04T09:22:55.3831130Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3832012Z raise RuntimeError(msg) 2025-12-04T09:22:55.3833582Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.3835091Z 2025-12-04T09:22:55.3835314Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3836508Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3837477Z 2025-12-04T09:22:55.3837759Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3838163Z 2025-12-04T09:22:55.3838173Z 2025-12-04T09:22:55.3838405Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.3839045Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.3840262Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-cc80cfecc5abaf73.xml - 2025-12-04T09:22:55.3841254Z =========================== short test summary info ============================ 2025-12-04T09:22:55.3842403Z FAILED [8.5972s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.3843498Z Traceback (most recent call last): 2025-12-04T09:22:55.3844208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3844932Z getattr(self, test_name)() 2025-12-04T09:22:55.3845595Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3846286Z fn() 2025-12-04T09:22:55.3846868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3847533Z method(*args, **kwargs) 2025-12-04T09:22:55.3848169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3848846Z method(*args, **kwargs) 2025-12-04T09:22:55.3849481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3850139Z with policy(): 2025-12-04T09:22:55.3850748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3851439Z raise RuntimeError(msg) 2025-12-04T09:22:55.3852914Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.3854230Z 2025-12-04T09:22:55.3854424Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3855475Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3856335Z 2025-12-04T09:22:55.3856742Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3857301Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.3857763Z ======================= 1 failed, 9 deselected in 8.62s ======================== 2025-12-04T09:22:55.3858234Z Got exit code 1 2025-12-04T09:22:55.3858492Z Retrying single test... 2025-12-04T09:22:55.3859249Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-00aad17389ddcea3.xml 2025-12-04T09:22:55.3860127Z ============================= test session starts ============================== 2025-12-04T09:22:55.3860748Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.3861319Z cachedir: .pytest_cache 2025-12-04T09:22:55.3861977Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.3862712Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.3863050Z configfile: pytest.ini 2025-12-04T09:22:55.3863735Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.3864563Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.3865748Z stepcurrent: skipping 0 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3866828Z Running 1 items in this shard 2025-12-04T09:22:55.3867024Z 2025-12-04T09:22:55.3868468Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda I1204 09:16:49.664000 29299 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 29351 2025-12-04T09:22:55.3870269Z I1204 09:16:49.665000 29299 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 29352 2025-12-04T09:22:55.3871408Z I1204 09:16:49.666000 29299 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 29353 2025-12-04T09:22:55.3872547Z I1204 09:16:49.666000 29299 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 29354 2025-12-04T09:22:55.3874426Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3875918Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3877881Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3880013Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3881577Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3882923Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3884230Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3885565Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3887309Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3889153Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3890937Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3892717Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3894072Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.3895409Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.3897164Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.3898941Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.3899611Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3900620Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3902121Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3903588Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3905053Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3906412Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3908013Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3909680Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3911280Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3912881Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3914463Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3916027Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3917594Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3919252Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3921611Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.3923794Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3925119Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3927207Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3928995Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3930222Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3931625Z [rank1]:E1204 09:16:56.415000 29352 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.3932774Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3933908Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3935586Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3937351Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3938811Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3940175Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3941628Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3943051Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3944453Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3945864Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3947358Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3949155Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3950724Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3952316Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3954781Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.3957090Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3958261Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3960411Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3961985Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3963062Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3964325Z [rank2]:E1204 09:16:56.456000 29353 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.3965341Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3966347Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.3967825Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.3969289Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.3970795Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.3972164Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.3973503Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3974914Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3976332Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.3977755Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.3979228Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.3980589Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.3981973Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.3983400Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.3985565Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 485425152 and is now 634322944. 2025-12-04T09:22:55.3987842Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3989009Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.3991077Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.3992852Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.3994085Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.3995491Z [rank3]:E1204 09:16:56.469000 29354 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.3996276Z dist init r=1, world=4 2025-12-04T09:22:55.3996562Z dist init r=2, world=4 2025-12-04T09:22:55.3996849Z dist init r=3, world=4 2025-12-04T09:22:55.3997487Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.3998606Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4000519Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4001991Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4003450Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4004794Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4006136Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4007555Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4009029Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4010442Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4012070Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4013523Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4014997Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4016506Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4018806Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.4020939Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4022021Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4024347Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.4026112Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4027405Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4028787Z [rank0]:E1204 09:16:56.646000 29351 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.4029559Z dist init r=0, world=4 2025-12-04T09:22:55.4030975Z [rank0]:[W1204 09:16:57.982096653 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.4032350Z FAILED [8.6283s] [100%] 2025-12-04T09:22:55.4032531Z 2025-12-04T09:22:55.4032683Z =================================== FAILURES =================================== 2025-12-04T09:22:55.4033409Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda _ 2025-12-04T09:22:55.4034091Z Traceback (most recent call last): 2025-12-04T09:22:55.4034865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.4035645Z self._join_processes(fn) 2025-12-04T09:22:55.4036427Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.4037281Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.4038243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.4039081Z raise RuntimeError(error) 2025-12-04T09:22:55.4039594Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.4040017Z Traceback (most recent call last): 2025-12-04T09:22:55.4040694Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4041384Z getattr(self, test_name)() 2025-12-04T09:22:55.4042030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4042699Z fn() 2025-12-04T09:22:55.4043253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4043902Z method(*args, **kwargs) 2025-12-04T09:22:55.4044520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4045173Z method(*args, **kwargs) 2025-12-04T09:22:55.4045796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4046440Z with policy(): 2025-12-04T09:22:55.4047025Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4047682Z raise RuntimeError(msg) 2025-12-04T09:22:55.4049047Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.4050353Z 2025-12-04T09:22:55.4050543Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4051576Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.4052418Z 2025-12-04T09:22:55.4052648Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4052997Z 2025-12-04T09:22:55.4053001Z 2025-12-04T09:22:55.4053201Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.4053738Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.4054782Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-00aad17389ddcea3.xml - 2025-12-04T09:22:55.4055751Z =========================== short test summary info ============================ 2025-12-04T09:22:55.4056942Z FAILED [8.6283s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.4058013Z Traceback (most recent call last): 2025-12-04T09:22:55.4058700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4059401Z getattr(self, test_name)() 2025-12-04T09:22:55.4060056Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4060721Z fn() 2025-12-04T09:22:55.4061288Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4061946Z method(*args, **kwargs) 2025-12-04T09:22:55.4062561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4063311Z method(*args, **kwargs) 2025-12-04T09:22:55.4063925Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4064575Z with policy(): 2025-12-04T09:22:55.4065160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4065821Z raise RuntimeError(msg) 2025-12-04T09:22:55.4067189Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.4068846Z 2025-12-04T09:22:55.4069064Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4070243Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.4071189Z 2025-12-04T09:22:55.4071450Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4072025Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.4072510Z ======================= 1 failed, 9 deselected in 8.65s ======================== 2025-12-04T09:22:55.4072911Z Got exit code 1 2025-12-04T09:22:55.4073811Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.4075079Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.4076235Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc7285fa7f4227.xml 2025-12-04T09:22:55.4077142Z ============================= test session starts ============================== 2025-12-04T09:22:55.4077786Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.4078364Z cachedir: .pytest_cache 2025-12-04T09:22:55.4079055Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.4079933Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.4080239Z configfile: pytest.ini 2025-12-04T09:22:55.4080874Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.4081642Z collecting ... collected 10 items / 1 deselected / 9 selected 2025-12-04T09:22:55.4082067Z stepcurrent: skipping 1 already run items. 2025-12-04T09:22:55.4082402Z Running 9 items in this shard 2025-12-04T09:22:55.4082586Z 2025-12-04T09:22:55.4083712Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda I1204 09:17:03.194000 29636 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 29688 2025-12-04T09:22:55.4085310Z I1204 09:17:03.195000 29636 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 29689 2025-12-04T09:22:55.4086303Z I1204 09:17:03.195000 29636 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 29690 2025-12-04T09:22:55.4087301Z I1204 09:17:03.196000 29636 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 29691 2025-12-04T09:22:55.4088960Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4090347Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4092077Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4093850Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4095203Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4096522Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4098255Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4100014Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4101366Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4102679Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4104405Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4106176Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4107583Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4109229Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4111241Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4113253Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4114009Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4115122Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4116799Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4118435Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4120173Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4121581Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4122896Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4124686Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4126271Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4127873Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4129453Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4130990Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4132537Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4134131Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4136773Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 464453632 and is now 634322944. 2025-12-04T09:22:55.4138935Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4140016Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4141972Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4143765Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4144918Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4146218Z [rank3]:E1204 09:17:09.922000 29691 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.4147332Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4148594Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4150254Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4151994Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4153618Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4155133Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4156622Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4158205Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4159899Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4161429Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4162829Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4164189Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4165551Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4166966Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4169113Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.4171145Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4172159Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4174042Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4175619Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4176685Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4177924Z [rank2]:E1204 09:17:09.927000 29690 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.4178924Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4179913Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4181432Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4182885Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4184335Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4185688Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4187016Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4188754Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4190331Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4191904Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4193488Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4195023Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4196575Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4198178Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4200661Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.4202685Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4203771Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4205602Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4207169Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4208246Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4209482Z [rank1]:E1204 09:17:10.033000 29689 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.4210241Z dist init r=3, world=4 2025-12-04T09:22:55.4210482Z dist init r=2, world=4 2025-12-04T09:22:55.4210719Z dist init r=1, world=4 2025-12-04T09:22:55.4211273Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4212252Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4213725Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4215170Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4216619Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4217963Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4219280Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4220678Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4222082Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4223477Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4225371Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4226907Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4228538Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4230128Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4232671Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.4234960Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4236103Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4238170Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4240092Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4241226Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4242472Z [rank0]:E1204 09:17:10.227000 29688 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.4243157Z dist init r=0, world=4 2025-12-04T09:22:55.4244337Z [rank0]:[W1204 09:17:10.595392475 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.4245583Z FAILED [8.5204s] [ 11%] 2025-12-04T09:22:55.4245744Z 2025-12-04T09:22:55.4245895Z =================================== FAILURES =================================== 2025-12-04T09:22:55.4246556Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda _ 2025-12-04T09:22:55.4247192Z Traceback (most recent call last): 2025-12-04T09:22:55.4247896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.4248598Z self._join_processes(fn) 2025-12-04T09:22:55.4249313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.4250095Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.4250887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.4251645Z raise RuntimeError(error) 2025-12-04T09:22:55.4252058Z RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.4252507Z Traceback (most recent call last): 2025-12-04T09:22:55.4253202Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4253924Z getattr(self, test_name)() 2025-12-04T09:22:55.4254598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4255290Z fn() 2025-12-04T09:22:55.4255857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4256536Z method(*args, **kwargs) 2025-12-04T09:22:55.4257178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4257840Z method(*args, **kwargs) 2025-12-04T09:22:55.4258479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4259152Z with policy(): 2025-12-04T09:22:55.4259824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4260499Z raise RuntimeError(msg) 2025-12-04T09:22:55.4262096Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 464453632 and is now 634322944. 2025-12-04T09:22:55.4263493Z 2025-12-04T09:22:55.4263697Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4264809Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4265705Z 2025-12-04T09:22:55.4265969Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4266400Z 2025-12-04T09:22:55.4266409Z 2025-12-04T09:22:55.4266620Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.4267272Z Process 3 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.4268632Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc7285fa7f4227.xml - 2025-12-04T09:22:55.4269747Z =========================== short test summary info ============================ 2025-12-04T09:22:55.4271041Z FAILED [8.5204s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda - RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.4272386Z Traceback (most recent call last): 2025-12-04T09:22:55.4273244Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4274057Z getattr(self, test_name)() 2025-12-04T09:22:55.4274800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4275583Z fn() 2025-12-04T09:22:55.4276231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4276971Z method(*args, **kwargs) 2025-12-04T09:22:55.4277685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4278451Z method(*args, **kwargs) 2025-12-04T09:22:55.4279161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4280006Z with policy(): 2025-12-04T09:22:55.4280664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4281415Z raise RuntimeError(msg) 2025-12-04T09:22:55.4282911Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 464453632 and is now 634322944. 2025-12-04T09:22:55.4284350Z 2025-12-04T09:22:55.4284560Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4285791Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4286696Z 2025-12-04T09:22:55.4286948Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4287506Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.4288045Z ======================= 1 failed, 1 deselected in 8.55s ======================== 2025-12-04T09:22:55.4288453Z Got exit code 1 2025-12-04T09:22:55.4288714Z Retrying single test... 2025-12-04T09:22:55.4289472Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-69d9e5e2bc4a3a7d.xml 2025-12-04T09:22:55.4290349Z ============================= test session starts ============================== 2025-12-04T09:22:55.4290975Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.4291542Z cachedir: .pytest_cache 2025-12-04T09:22:55.4292371Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.4293125Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.4293468Z configfile: pytest.ini 2025-12-04T09:22:55.4294166Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.4295085Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.4296302Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4297416Z Running 1 items in this shard 2025-12-04T09:22:55.4297622Z 2025-12-04T09:22:55.4298774Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda I1204 09:17:16.634000 29973 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 30025 2025-12-04T09:22:55.4300527Z I1204 09:17:16.635000 29973 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 30026 2025-12-04T09:22:55.4301635Z I1204 09:17:16.636000 29973 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 30027 2025-12-04T09:22:55.4302731Z I1204 09:17:16.636000 29973 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 30028 2025-12-04T09:22:55.4304864Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4306185Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4308214Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4310235Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4311771Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4313279Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4314751Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4316257Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4318769Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4320926Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4322705Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4324896Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4326442Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4328048Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4330007Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4332012Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4332760Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4333891Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4335574Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4337300Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4338754Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4340094Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4341433Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4342853Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4344273Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4345687Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4347090Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4348810Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4350460Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4352063Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4354524Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.4356813Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4358042Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4360324Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4361918Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4363013Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4364247Z [rank1]:E1204 09:17:23.388000 30026 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.4365279Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4366287Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4367780Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4369234Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4370697Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4372067Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4373423Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4374832Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4376236Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4377651Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4379133Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4380501Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4382052Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4383531Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4385808Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.4388439Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4389588Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4391649Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4393399Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4394604Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4396004Z [rank2]:E1204 09:17:23.480000 30027 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.4397128Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4398236Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4400102Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4401554Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4403001Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4404341Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4405657Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4407057Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4408455Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4409911Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4411309Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4412666Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4414040Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4415445Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4417590Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 581894144 and is now 634322944. 2025-12-04T09:22:55.4419665Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4420685Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4422525Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4424501Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4425717Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4427110Z [rank3]:E1204 09:17:23.483000 30028 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.4427955Z dist init r=1, world=4 2025-12-04T09:22:55.4428225Z dist init r=2, world=4 2025-12-04T09:22:55.4428487Z dist init r=3, world=4 2025-12-04T09:22:55.4429108Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4430220Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4431887Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4433512Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4435143Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4436650Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4438144Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4439886Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4441282Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4442670Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4444061Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4445426Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4446861Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4448261Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4450402Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.4452415Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4453434Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4455269Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4456831Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4457903Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4459122Z [rank0]:E1204 09:17:23.621000 30025 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.4459810Z dist init r=0, world=4 2025-12-04T09:22:55.4460991Z [rank0]:[W1204 09:17:24.964834812 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.4462206Z FAILED [8.8048s] [100%] 2025-12-04T09:22:55.4462360Z 2025-12-04T09:22:55.4462488Z =================================== FAILURES =================================== 2025-12-04T09:22:55.4463127Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda _ 2025-12-04T09:22:55.4463737Z Traceback (most recent call last): 2025-12-04T09:22:55.4464420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.4465105Z self._join_processes(fn) 2025-12-04T09:22:55.4465791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.4466608Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.4467430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.4468424Z raise RuntimeError(error) 2025-12-04T09:22:55.4468871Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.4469358Z Traceback (most recent call last): 2025-12-04T09:22:55.4470132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4470938Z getattr(self, test_name)() 2025-12-04T09:22:55.4471693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4472450Z fn() 2025-12-04T09:22:55.4473094Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4473928Z method(*args, **kwargs) 2025-12-04T09:22:55.4474641Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4475388Z method(*args, **kwargs) 2025-12-04T09:22:55.4476098Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4476848Z with policy(): 2025-12-04T09:22:55.4477518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4478286Z raise RuntimeError(msg) 2025-12-04T09:22:55.4479929Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.4481372Z 2025-12-04T09:22:55.4481580Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4482831Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4483731Z 2025-12-04T09:22:55.4483980Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4484368Z 2025-12-04T09:22:55.4484523Z Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.4484905Z Traceback (most recent call last): 2025-12-04T09:22:55.4485644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4486387Z getattr(self, test_name)() 2025-12-04T09:22:55.4487092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4487823Z fn() 2025-12-04T09:22:55.4488422Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4489134Z method(*args, **kwargs) 2025-12-04T09:22:55.4489810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4490524Z method(*args, **kwargs) 2025-12-04T09:22:55.4491177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4491883Z with policy(): 2025-12-04T09:22:55.4492520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4493239Z raise RuntimeError(msg) 2025-12-04T09:22:55.4494880Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.4496213Z 2025-12-04T09:22:55.4496408Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4497458Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4498308Z 2025-12-04T09:22:55.4498559Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4498913Z 2025-12-04T09:22:55.4498917Z 2025-12-04T09:22:55.4499121Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.4499679Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.4500808Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-69d9e5e2bc4a3a7d.xml - 2025-12-04T09:22:55.4501793Z =========================== short test summary info ============================ 2025-12-04T09:22:55.4502941Z FAILED [8.8048s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.4504037Z Traceback (most recent call last): 2025-12-04T09:22:55.4504742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4505456Z getattr(self, test_name)() 2025-12-04T09:22:55.4506113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4506801Z fn() 2025-12-04T09:22:55.4507456Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4508355Z method(*args, **kwargs) 2025-12-04T09:22:55.4509074Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4509842Z method(*args, **kwargs) 2025-12-04T09:22:55.4510561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4511301Z with policy(): 2025-12-04T09:22:55.4511986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4512752Z raise RuntimeError(msg) 2025-12-04T09:22:55.4514316Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.4515798Z 2025-12-04T09:22:55.4516016Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4517195Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4518162Z 2025-12-04T09:22:55.4518427Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4518830Z 2025-12-04T09:22:55.4519008Z Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.4519420Z Traceback (most recent call last): 2025-12-04T09:22:55.4521281Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4521995Z getattr(self, test_name)() 2025-12-04T09:22:55.4522742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4523416Z fn() 2025-12-04T09:22:55.4524336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4525097Z method(*args, **kwargs) 2025-12-04T09:22:55.4525797Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4526554Z method(*args, **kwargs) 2025-12-04T09:22:55.4527266Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4528014Z with policy(): 2025-12-04T09:22:55.4528682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4529448Z raise RuntimeError(msg) 2025-12-04T09:22:55.4531162Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.4532638Z 2025-12-04T09:22:55.4532869Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4534036Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4535011Z 2025-12-04T09:22:55.4535281Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4535876Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.4536492Z ======================= 1 failed, 9 deselected in 8.83s ======================== 2025-12-04T09:22:55.4537001Z Got exit code 1 2025-12-04T09:22:55.4537258Z Retrying single test... 2025-12-04T09:22:55.4537982Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c1c17128b5541633.xml 2025-12-04T09:22:55.4538807Z ============================= test session starts ============================== 2025-12-04T09:22:55.4539388Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.4539923Z cachedir: .pytest_cache 2025-12-04T09:22:55.4540547Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.4541229Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.4541544Z configfile: pytest.ini 2025-12-04T09:22:55.4542192Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.4542988Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.4544099Z stepcurrent: skipping 1 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4545120Z Running 1 items in this shard 2025-12-04T09:22:55.4545308Z 2025-12-04T09:22:55.4546369Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda I1204 09:17:30.084000 30310 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 30362 2025-12-04T09:22:55.4548321Z I1204 09:17:30.085000 30310 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 30363 2025-12-04T09:22:55.4549454Z I1204 09:17:30.085000 30310 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 30364 2025-12-04T09:22:55.4550684Z I1204 09:17:30.086000 30310 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 30365 2025-12-04T09:22:55.4552573Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4554090Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4556074Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4558079Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4559786Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4561272Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4562595Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4563929Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4565670Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4567461Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4569251Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4571037Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4572395Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4573724Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4575463Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4577246Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4577921Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4578926Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4580454Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4581931Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4583388Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4584754Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4586094Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4587560Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4589403Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4590992Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4592592Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4594134Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4595711Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4597320Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4599876Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 598671360 and is now 634322944. 2025-12-04T09:22:55.4602050Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4603090Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4604933Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4606513Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4607605Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4608856Z [rank1]:E1204 09:17:36.716000 30363 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.4609855Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4610908Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4612395Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4613860Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4615317Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4616670Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4618011Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4619485Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4620904Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4622317Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4623912Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4625675Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4627322Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4628929Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4631393Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.4633714Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4634881Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4636963Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4638744Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4640133Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4641543Z [rank2]:E1204 09:17:36.845000 30364 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.4642632Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4643702Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4645278Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4646818Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4648367Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4649878Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4651288Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4652787Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4654272Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4655773Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4657337Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4658725Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4660113Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4661521Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4663693Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 495910912 and is now 634322944. 2025-12-04T09:22:55.4665746Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4666786Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4668991Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4670758Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4672058Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4673463Z [rank3]:E1204 09:17:36.852000 30365 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.4674261Z dist init r=1, world=4 2025-12-04T09:22:55.4674534Z dist init r=2, world=4 2025-12-04T09:22:55.4675173Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4676304Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4677984Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4679787Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4681387Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4682738Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4684067Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4685476Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4686880Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4688289Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4689695Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4691064Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4692439Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4693855Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4696011Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.4698047Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4699078Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4700991Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4702565Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4703643Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4704881Z [rank0]:E1204 09:17:37.001000 30362 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.4704974Z dist init r=0, world=4 2025-12-04T09:22:55.4705071Z dist init r=3, world=4 2025-12-04T09:22:55.4706265Z [rank0]:[W1204 09:17:37.323784900 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.4706421Z FAILED [8.3014s] [100%] 2025-12-04T09:22:55.4706427Z 2025-12-04T09:22:55.4706567Z =================================== FAILURES =================================== 2025-12-04T09:22:55.4706982Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda _ 2025-12-04T09:22:55.4707105Z Traceback (most recent call last): 2025-12-04T09:22:55.4707844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.4707971Z self._join_processes(fn) 2025-12-04T09:22:55.4708554Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.4708695Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.4709322Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.4709437Z raise RuntimeError(error) 2025-12-04T09:22:55.4709671Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.4709799Z Traceback (most recent call last): 2025-12-04T09:22:55.4710341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4710469Z getattr(self, test_name)() 2025-12-04T09:22:55.4711001Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4711089Z fn() 2025-12-04T09:22:55.4711603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4711705Z method(*args, **kwargs) 2025-12-04T09:22:55.4712215Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4712324Z method(*args, **kwargs) 2025-12-04T09:22:55.4712821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4712930Z with policy(): 2025-12-04T09:22:55.4713437Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4713545Z raise RuntimeError(msg) 2025-12-04T09:22:55.4714905Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 598671360 and is now 634322944. 2025-12-04T09:22:55.4714911Z 2025-12-04T09:22:55.4715121Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4716018Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4716025Z 2025-12-04T09:22:55.4716291Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4716297Z 2025-12-04T09:22:55.4716301Z 2025-12-04T09:22:55.4716533Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.4716798Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.4717590Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c1c17128b5541633.xml - 2025-12-04T09:22:55.4717771Z =========================== short test summary info ============================ 2025-12-04T09:22:55.4718751Z FAILED [8.3014s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.4718938Z Traceback (most recent call last): 2025-12-04T09:22:55.4719485Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4719701Z getattr(self, test_name)() 2025-12-04T09:22:55.4720331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4720409Z fn() 2025-12-04T09:22:55.4720857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4720961Z method(*args, **kwargs) 2025-12-04T09:22:55.4721405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4721513Z method(*args, **kwargs) 2025-12-04T09:22:55.4721960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4722045Z with policy(): 2025-12-04T09:22:55.4722501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4722597Z raise RuntimeError(msg) 2025-12-04T09:22:55.4723948Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 598671360 and is now 634322944. 2025-12-04T09:22:55.4723954Z 2025-12-04T09:22:55.4724146Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4725128Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4725137Z 2025-12-04T09:22:55.4725410Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4725587Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.4725767Z ======================= 1 failed, 9 deselected in 8.32s ======================== 2025-12-04T09:22:55.4725864Z Got exit code 1 2025-12-04T09:22:55.4726610Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.4727029Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.4727643Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-156a51594e715e30.xml 2025-12-04T09:22:55.4727912Z ============================= test session starts ============================== 2025-12-04T09:22:55.4728262Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.4728372Z cachedir: .pytest_cache 2025-12-04T09:22:55.4728896Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.4729019Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.4729127Z configfile: pytest.ini 2025-12-04T09:22:55.4729669Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.4729879Z collecting ... collected 10 items / 2 deselected / 8 selected 2025-12-04T09:22:55.4730028Z stepcurrent: skipping 2 already run items. 2025-12-04T09:22:55.4730141Z Running 8 items in this shard 2025-12-04T09:22:55.4730146Z 2025-12-04T09:22:55.4731409Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda I1204 09:17:43.544000 30647 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 30699 2025-12-04T09:22:55.4731915Z I1204 09:17:43.545000 30647 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 30700 2025-12-04T09:22:55.4732399Z I1204 09:17:43.546000 30647 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 30701 2025-12-04T09:22:55.4732899Z I1204 09:17:43.546000 30647 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 30702 2025-12-04T09:22:55.4734141Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4734285Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4735512Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4735637Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4737405Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4737552Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4739087Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4739234Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4740338Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4740450Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4742006Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4742164Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4743250Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4743377Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4744897Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4745098Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4745506Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4745981Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4746874Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4747377Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4748514Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4748916Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4749889Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4750377Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4751339Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4751837Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4752797Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4753251Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4754211Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4754718Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4756595Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.4756967Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4757628Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4758906Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4759277Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4760152Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4760640Z [rank2]:E1204 09:17:50.399000 30701 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.4761040Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4761511Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4762400Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4762857Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4763740Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4764088Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4764947Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4765377Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4766228Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4766675Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4767517Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4767925Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4768772Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4769271Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4770856Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.4771183Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4771768Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4772891Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4773270Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4773905Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4774387Z [rank1]:E1204 09:17:50.402000 30700 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.4774789Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4775253Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4776149Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4776595Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4788006Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4788479Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4789460Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4789972Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4790939Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4791436Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4792390Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4792841Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4793936Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4794423Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4796247Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 581894144 and is now 634322944. 2025-12-04T09:22:55.4796613Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4797348Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4798619Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4798988Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4799704Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4800333Z [rank3]:E1204 09:17:50.410000 30702 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.4800429Z dist init r=2, world=4 2025-12-04T09:22:55.4800518Z dist init r=1, world=4 2025-12-04T09:22:55.4800614Z dist init r=3, world=4 2025-12-04T09:22:55.4801014Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4801482Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4802382Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4802831Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4803715Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4804068Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4804923Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4805352Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4806199Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4806636Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4807534Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4807937Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4808788Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4809234Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4810828Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.4811217Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4811813Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4812937Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4813264Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4813904Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4814394Z [rank0]:E1204 09:17:50.530000 30699 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.4814487Z dist init r=0, world=4 2025-12-04T09:22:55.4815513Z [rank0]:[W1204 09:17:50.851528979 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.4815613Z FAILED [9.2623s] [ 12%] 2025-12-04T09:22:55.4815620Z 2025-12-04T09:22:55.4815754Z =================================== FAILURES =================================== 2025-12-04T09:22:55.4816153Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda _ 2025-12-04T09:22:55.4816271Z Traceback (most recent call last): 2025-12-04T09:22:55.4816761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.4816868Z self._join_processes(fn) 2025-12-04T09:22:55.4817382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.4817507Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.4818055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.4818154Z raise RuntimeError(error) 2025-12-04T09:22:55.4818361Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.4818476Z Traceback (most recent call last): 2025-12-04T09:22:55.4819004Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4819116Z getattr(self, test_name)() 2025-12-04T09:22:55.4819586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4819663Z fn() 2025-12-04T09:22:55.4820118Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4820209Z method(*args, **kwargs) 2025-12-04T09:22:55.4820654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4820750Z method(*args, **kwargs) 2025-12-04T09:22:55.4821192Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4821286Z with policy(): 2025-12-04T09:22:55.4821738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4821882Z raise RuntimeError(msg) 2025-12-04T09:22:55.4823079Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.4823085Z 2025-12-04T09:22:55.4823276Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4824371Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4824379Z 2025-12-04T09:22:55.4824649Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4824655Z 2025-12-04T09:22:55.4824665Z 2025-12-04T09:22:55.4824897Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.4825157Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.4825950Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-156a51594e715e30.xml - 2025-12-04T09:22:55.4826129Z =========================== short test summary info ============================ 2025-12-04T09:22:55.4827109Z FAILED [9.2623s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.4827318Z Traceback (most recent call last): 2025-12-04T09:22:55.4827879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4827996Z getattr(self, test_name)() 2025-12-04T09:22:55.4828542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4828637Z fn() 2025-12-04T09:22:55.4829142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4829260Z method(*args, **kwargs) 2025-12-04T09:22:55.4829763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4829874Z method(*args, **kwargs) 2025-12-04T09:22:55.4830376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4830472Z with policy(): 2025-12-04T09:22:55.4830996Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4831109Z raise RuntimeError(msg) 2025-12-04T09:22:55.4832577Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 607059968 and is now 634322944. 2025-12-04T09:22:55.4832584Z 2025-12-04T09:22:55.4832799Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4833616Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4833632Z 2025-12-04T09:22:55.4833897Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4834079Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.4834263Z ======================= 1 failed, 2 deselected in 9.29s ======================== 2025-12-04T09:22:55.4834442Z Got exit code 1 2025-12-04T09:22:55.4834550Z Retrying single test... 2025-12-04T09:22:55.4835173Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2e6a295727102ba7.xml 2025-12-04T09:22:55.4835336Z ============================= test session starts ============================== 2025-12-04T09:22:55.4835693Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.4835802Z cachedir: .pytest_cache 2025-12-04T09:22:55.4836320Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.4836447Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.4836555Z configfile: pytest.ini 2025-12-04T09:22:55.4837085Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.4837308Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.4838206Z stepcurrent: skipping 2 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4838327Z Running 1 items in this shard 2025-12-04T09:22:55.4838333Z 2025-12-04T09:22:55.4839602Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda I1204 09:17:57.474000 30984 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 31036 2025-12-04T09:22:55.4840042Z I1204 09:17:57.475000 30984 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 31037 2025-12-04T09:22:55.4840484Z I1204 09:17:57.476000 30984 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 31038 2025-12-04T09:22:55.4840928Z I1204 09:17:57.477000 30984 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 31039 2025-12-04T09:22:55.4842041Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4842153Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4843443Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4843564Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4844807Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4844938Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4846546Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4846717Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4848525Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4848751Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4850402Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4850570Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4851760Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4851891Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4853546Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4853703Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4854155Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4854670Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4855653Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4856154Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4857113Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4857513Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4858444Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4858981Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4859903Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4860372Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4861304Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4861731Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4862676Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4863283Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4865032Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.4865384Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4866030Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4867328Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4867685Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4868568Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4869113Z [rank2]:E1204 09:18:04.342000 31038 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.4869566Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4870101Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4871102Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4871614Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4872668Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4873071Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4874256Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4874754Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4875846Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4876374Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4877423Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4878479Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4879593Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4880251Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4882145Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.4882566Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4883254Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4884572Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4884992Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4885760Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4886443Z [rank1]:E1204 09:18:04.342000 31037 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.4886852Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4887383Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4888340Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4888852Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4889854Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4890285Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4891195Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4891692Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4892815Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4893351Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4894375Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4894857Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4895834Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4896358Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4898190Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 485425152 and is now 634322944. 2025-12-04T09:22:55.4898601Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4899290Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4900558Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4900964Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4901725Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4902280Z [rank3]:E1204 09:18:04.351000 31039 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.4902402Z dist init r=3, world=4 2025-12-04T09:22:55.4902499Z dist init r=1, world=4 2025-12-04T09:22:55.4902588Z dist init r=2, world=4 2025-12-04T09:22:55.4903076Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4903698Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4904759Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4905215Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4906207Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4906570Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4907571Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4908254Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4909399Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4909956Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4910989Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4911436Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4912484Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4913041Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4914843Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.4915275Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4915939Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4917221Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4917593Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4918303Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4918851Z [rank0]:E1204 09:18:04.515000 31036 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.4919001Z dist init r=0, world=4 2025-12-04T09:22:55.4920418Z [rank0]:[W1204 09:18:04.849394929 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.4920525Z FAILED [9.1888s] [100%] 2025-12-04T09:22:55.4920531Z 2025-12-04T09:22:55.4920665Z =================================== FAILURES =================================== 2025-12-04T09:22:55.4921055Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda _ 2025-12-04T09:22:55.4921178Z Traceback (most recent call last): 2025-12-04T09:22:55.4921660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.4921761Z self._join_processes(fn) 2025-12-04T09:22:55.4922283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.4922407Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.4923006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.4923105Z raise RuntimeError(error) 2025-12-04T09:22:55.4923312Z RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.4923422Z Traceback (most recent call last): 2025-12-04T09:22:55.4924253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4924373Z getattr(self, test_name)() 2025-12-04T09:22:55.4924903Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4924990Z fn() 2025-12-04T09:22:55.4925497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4925608Z method(*args, **kwargs) 2025-12-04T09:22:55.4926196Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4926306Z method(*args, **kwargs) 2025-12-04T09:22:55.4926813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4926914Z with policy(): 2025-12-04T09:22:55.4927418Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4927526Z raise RuntimeError(msg) 2025-12-04T09:22:55.4928878Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 485425152 and is now 634322944. 2025-12-04T09:22:55.4928888Z 2025-12-04T09:22:55.4929109Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4929939Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4929945Z 2025-12-04T09:22:55.4930210Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4930215Z 2025-12-04T09:22:55.4930220Z 2025-12-04T09:22:55.4930447Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.4930708Z Process 3 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.4931506Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2e6a295727102ba7.xml - 2025-12-04T09:22:55.4931682Z =========================== short test summary info ============================ 2025-12-04T09:22:55.4932769Z FAILED [9.1888s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda - RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.4932898Z Traceback (most recent call last): 2025-12-04T09:22:55.4933448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4933559Z getattr(self, test_name)() 2025-12-04T09:22:55.4934100Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4934188Z fn() 2025-12-04T09:22:55.4934698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4934809Z method(*args, **kwargs) 2025-12-04T09:22:55.4935313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4935501Z method(*args, **kwargs) 2025-12-04T09:22:55.4936002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4936100Z with policy(): 2025-12-04T09:22:55.4936703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4936800Z raise RuntimeError(msg) 2025-12-04T09:22:55.4937988Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 485425152 and is now 634322944. 2025-12-04T09:22:55.4938000Z 2025-12-04T09:22:55.4938189Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4938922Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4938927Z 2025-12-04T09:22:55.4939167Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4939394Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.4939562Z ======================= 1 failed, 9 deselected in 9.21s ======================== 2025-12-04T09:22:55.4939649Z Got exit code 1 2025-12-04T09:22:55.4939744Z Retrying single test... 2025-12-04T09:22:55.4940300Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2bb3299d57ce6c2b.xml 2025-12-04T09:22:55.4940444Z ============================= test session starts ============================== 2025-12-04T09:22:55.4940753Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.4940864Z cachedir: .pytest_cache 2025-12-04T09:22:55.4941323Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.4941437Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.4941532Z configfile: pytest.ini 2025-12-04T09:22:55.4942008Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.4942203Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.4942999Z stepcurrent: skipping 2 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4943107Z Running 1 items in this shard 2025-12-04T09:22:55.4943112Z 2025-12-04T09:22:55.4944219Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda I1204 09:18:11.414000 31321 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 31373 2025-12-04T09:22:55.4944663Z I1204 09:18:11.415000 31321 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 31374 2025-12-04T09:22:55.4945099Z I1204 09:18:11.416000 31321 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 31375 2025-12-04T09:22:55.4945531Z I1204 09:18:11.416000 31321 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 31376 2025-12-04T09:22:55.4946640Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4946754Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4948667Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4948837Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4950074Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4950214Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4951449Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4951589Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4953298Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4953478Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4955184Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4955350Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4956585Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.4956708Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.4958440Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.4958665Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.4959132Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4959765Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4960798Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4961253Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4962125Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4962535Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4963381Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4963819Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4964666Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4965097Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4966035Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4966430Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4967294Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4967726Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4969336Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 577699840 and is now 634322944. 2025-12-04T09:22:55.4969658Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4970239Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4971374Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4971697Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4972418Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4972897Z [rank2]:E1204 09:18:18.193000 31375 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.4973001Z dist init r=2, world=4 2025-12-04T09:22:55.4973396Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4973863Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4974757Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4975258Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4976143Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4976493Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4977339Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4977776Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4978628Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4979073Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4979914Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4980317Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4981167Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4981604Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4983198Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.4983520Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4984110Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.4985289Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.4985624Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4986257Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.4986746Z [rank1]:E1204 09:18:18.316000 31374 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.4987143Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.4987678Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.4988925Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.4989431Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.4990420Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.4990821Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.4991775Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4992263Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4993224Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.4993706Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.4994667Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.4995110Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.4996085Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.4996571Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.4998366Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 588185600 and is now 634322944. 2025-12-04T09:22:55.4998737Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.4999443Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5000766Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.5001105Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5001873Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5002354Z [rank3]:E1204 09:18:18.320000 31376 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5002445Z dist init r=1, world=4 2025-12-04T09:22:55.5002588Z dist init r=3, world=4 2025-12-04T09:22:55.5002991Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5003464Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5004349Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5004795Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5005674Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5006034Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5006891Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5007323Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5008170Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5008597Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5009451Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5009847Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5010697Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5011134Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5012784Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.5013115Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5013693Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5014826Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.5015143Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5015776Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5016323Z [rank0]:E1204 09:18:18.440000 31373 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5016412Z dist init r=0, world=4 2025-12-04T09:22:55.5017518Z [rank0]:[W1204 09:18:18.780438052 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5017609Z FAILED [8.2867s] [100%] 2025-12-04T09:22:55.5017614Z 2025-12-04T09:22:55.5017743Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5018204Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda _ 2025-12-04T09:22:55.5018318Z Traceback (most recent call last): 2025-12-04T09:22:55.5018807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5018907Z self._join_processes(fn) 2025-12-04T09:22:55.5019423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5019557Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5020088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5020185Z raise RuntimeError(error) 2025-12-04T09:22:55.5020399Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5020504Z Traceback (most recent call last): 2025-12-04T09:22:55.5020984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5021089Z getattr(self, test_name)() 2025-12-04T09:22:55.5021559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5021646Z fn() 2025-12-04T09:22:55.5022092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5022182Z method(*args, **kwargs) 2025-12-04T09:22:55.5022634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5022731Z method(*args, **kwargs) 2025-12-04T09:22:55.5023176Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5023260Z with policy(): 2025-12-04T09:22:55.5024010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5024194Z raise RuntimeError(msg) 2025-12-04T09:22:55.5025706Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 577699840 and is now 634322944. 2025-12-04T09:22:55.5025713Z 2025-12-04T09:22:55.5025931Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5026753Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.5026758Z 2025-12-04T09:22:55.5027029Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5027034Z 2025-12-04T09:22:55.5027039Z 2025-12-04T09:22:55.5027326Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5027697Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5028506Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2bb3299d57ce6c2b.xml - 2025-12-04T09:22:55.5028674Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5029653Z FAILED [8.2867s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5029772Z Traceback (most recent call last): 2025-12-04T09:22:55.5030319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5030437Z getattr(self, test_name)() 2025-12-04T09:22:55.5030982Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5031072Z fn() 2025-12-04T09:22:55.5031582Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5031685Z method(*args, **kwargs) 2025-12-04T09:22:55.5032194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5032293Z method(*args, **kwargs) 2025-12-04T09:22:55.5032788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5032889Z with policy(): 2025-12-04T09:22:55.5033394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5033508Z raise RuntimeError(msg) 2025-12-04T09:22:55.5034865Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 577699840 and is now 634322944. 2025-12-04T09:22:55.5034871Z 2025-12-04T09:22:55.5035083Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5035907Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.5035912Z 2025-12-04T09:22:55.5036175Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5036357Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5036531Z ======================= 1 failed, 9 deselected in 8.31s ======================== 2025-12-04T09:22:55.5036630Z Got exit code 1 2025-12-04T09:22:55.5037443Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.5037846Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.5038471Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-b434b54a558efa89.xml 2025-12-04T09:22:55.5038631Z ============================= test session starts ============================== 2025-12-04T09:22:55.5038974Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5039087Z cachedir: .pytest_cache 2025-12-04T09:22:55.5039674Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5039843Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5039947Z configfile: pytest.ini 2025-12-04T09:22:55.5040422Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5040613Z collecting ... collected 10 items / 3 deselected / 7 selected 2025-12-04T09:22:55.5040739Z stepcurrent: skipping 3 already run items. 2025-12-04T09:22:55.5040834Z Running 7 items in this shard 2025-12-04T09:22:55.5040839Z 2025-12-04T09:22:55.5041896Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda I1204 09:18:25.004000 31658 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 31710 2025-12-04T09:22:55.5042338Z I1204 09:18:25.005000 31658 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 31711 2025-12-04T09:22:55.5042786Z I1204 09:18:25.006000 31658 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 31712 2025-12-04T09:22:55.5043220Z I1204 09:18:25.006000 31658 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 31713 2025-12-04T09:22:55.5044323Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5044435Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5045952Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5046112Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5047201Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5047322Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5048830Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5048980Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5050118Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5050232Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5051752Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5051892Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5052979Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5053135Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5054642Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5054783Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5055184Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5055664Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5056559Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5057016Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5057892Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5058248Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5059097Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5059536Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5060390Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5060821Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5061677Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5062072Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5063029Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5063463Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5065055Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.5065383Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5065965Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5067158Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5067541Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5068415Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5068960Z [rank1]:E1204 09:18:31.764000 31711 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5069060Z dist init r=1, world=4 2025-12-04T09:22:55.5069523Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5070047Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5071046Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5071550Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5072534Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5072947Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5073905Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5074394Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5075352Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5075843Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5076857Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5077305Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5078271Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5078757Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5080713Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.5081088Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5081679Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5082809Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5083135Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5083770Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5084254Z [rank2]:E1204 09:18:31.865000 31712 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5084659Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5085129Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5086018Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5086467Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5087350Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5087708Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5088557Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5088996Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5089836Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5090327Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5091171Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5091568Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5092430Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5092862Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5094523Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 586088448 and is now 634322944. 2025-12-04T09:22:55.5094845Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5095435Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5096555Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5096894Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5097527Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5098002Z [rank3]:E1204 09:18:31.889000 31713 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5098100Z dist init r=2, world=4 2025-12-04T09:22:55.5098186Z dist init r=3, world=4 2025-12-04T09:22:55.5098591Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5099059Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5099944Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5100403Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5101275Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5101635Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5102483Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5102965Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5103818Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5104248Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5105101Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5105496Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5106358Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5106845Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5108778Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.5109145Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5109806Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5111093Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5111456Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5112176Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5112722Z [rank0]:E1204 09:18:32.014000 31710 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5112831Z dist init r=0, world=4 2025-12-04T09:22:55.5113988Z [rank0]:[W1204 09:18:32.358127224 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5114092Z FAILED [8.5259s] [ 14%] 2025-12-04T09:22:55.5114098Z 2025-12-04T09:22:55.5114255Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5114696Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda _ 2025-12-04T09:22:55.5114822Z Traceback (most recent call last): 2025-12-04T09:22:55.5115371Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5115485Z self._join_processes(fn) 2025-12-04T09:22:55.5116077Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5116226Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5116904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5117024Z raise RuntimeError(error) 2025-12-04T09:22:55.5117260Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5117391Z Traceback (most recent call last): 2025-12-04T09:22:55.5117937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5118048Z getattr(self, test_name)() 2025-12-04T09:22:55.5118584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5118675Z fn() 2025-12-04T09:22:55.5119175Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5119343Z method(*args, **kwargs) 2025-12-04T09:22:55.5119941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5120045Z method(*args, **kwargs) 2025-12-04T09:22:55.5120514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5120607Z with policy(): 2025-12-04T09:22:55.5121092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5121196Z raise RuntimeError(msg) 2025-12-04T09:22:55.5122464Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.5122474Z 2025-12-04T09:22:55.5122680Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5123450Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5123456Z 2025-12-04T09:22:55.5123897Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5123904Z 2025-12-04T09:22:55.5123908Z 2025-12-04T09:22:55.5124289Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5124554Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5125355Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-b434b54a558efa89.xml - 2025-12-04T09:22:55.5125522Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5126507Z FAILED [8.5259s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5126627Z Traceback (most recent call last): 2025-12-04T09:22:55.5127178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5127287Z getattr(self, test_name)() 2025-12-04T09:22:55.5127821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5127917Z fn() 2025-12-04T09:22:55.5128420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5128529Z method(*args, **kwargs) 2025-12-04T09:22:55.5129125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5129231Z method(*args, **kwargs) 2025-12-04T09:22:55.5129738Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5129834Z with policy(): 2025-12-04T09:22:55.5130337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5130453Z raise RuntimeError(msg) 2025-12-04T09:22:55.5131800Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 604962816 and is now 634322944. 2025-12-04T09:22:55.5131806Z 2025-12-04T09:22:55.5132027Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5132943Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5132948Z 2025-12-04T09:22:55.5133218Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5133397Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5133573Z ======================= 1 failed, 3 deselected in 8.55s ======================== 2025-12-04T09:22:55.5133678Z Got exit code 1 2025-12-04T09:22:55.5133784Z Retrying single test... 2025-12-04T09:22:55.5134405Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-bd75c8cd2cf8ff9e.xml 2025-12-04T09:22:55.5134580Z ============================= test session starts ============================== 2025-12-04T09:22:55.5134928Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5135046Z cachedir: .pytest_cache 2025-12-04T09:22:55.5135554Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5135675Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5135791Z configfile: pytest.ini 2025-12-04T09:22:55.5136538Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5136722Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.5137533Z stepcurrent: skipping 3 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5137632Z Running 1 items in this shard 2025-12-04T09:22:55.5137637Z 2025-12-04T09:22:55.5138693Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda I1204 09:18:38.544000 31995 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 32047 2025-12-04T09:22:55.5139139Z I1204 09:18:38.545000 31995 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 32048 2025-12-04T09:22:55.5139581Z I1204 09:18:38.546000 31995 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 32049 2025-12-04T09:22:55.5140014Z I1204 09:18:38.546000 31995 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 32050 2025-12-04T09:22:55.5141116Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5141243Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5142810Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5142969Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5144058Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5144174Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5145697Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5145896Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5146987Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5147100Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5149007Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5149176Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5150408Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5150531Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5152234Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5152404Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5152865Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5153405Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5154402Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5154913Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5155958Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5156361Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5157327Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5157813Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5158782Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5159267Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5160378Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5160794Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5161694Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5162161Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5163846Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.5164196Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5164925Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5166061Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5166378Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5167024Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5167504Z [rank2]:E1204 09:18:45.215000 32049 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5167897Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5168373Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5169261Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5169784Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5170656Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5171002Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5171862Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5172293Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5173150Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5173628Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5174477Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5174873Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5175723Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5176170Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5177755Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 462356480 and is now 634322944. 2025-12-04T09:22:55.5178082Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5178661Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5179800Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5180126Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5180766Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5181246Z [rank3]:E1204 09:18:45.255000 32050 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5181337Z dist init r=2, world=4 2025-12-04T09:22:55.5181429Z dist init r=3, world=4 2025-12-04T09:22:55.5181830Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5182343Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5183236Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5183682Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5184556Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5184904Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5185770Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5186251Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5187098Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5187595Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5188717Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5189176Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5190141Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5190643Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5192430Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.5192806Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5193470Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5194743Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5195114Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5195830Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5196384Z [rank1]:E1204 09:18:45.369000 32048 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5196548Z dist init r=1, world=4 2025-12-04T09:22:55.5197000Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5197539Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5198539Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5199050Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5200133Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5200583Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5201512Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5201984Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5202923Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5203395Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5204341Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5204775Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5205720Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5206196Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5207943Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.5208409Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5209198Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5210444Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5210796Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5211557Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5212081Z [rank0]:E1204 09:18:45.506000 32047 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5212180Z dist init r=0, world=4 2025-12-04T09:22:55.5213314Z [rank0]:[W1204 09:18:45.841203813 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5213413Z FAILED [8.2736s] [100%] 2025-12-04T09:22:55.5213418Z 2025-12-04T09:22:55.5213571Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5213996Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda _ 2025-12-04T09:22:55.5214184Z Traceback (most recent call last): 2025-12-04T09:22:55.5214729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5214841Z self._join_processes(fn) 2025-12-04T09:22:55.5215414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5215555Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5216140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5216260Z raise RuntimeError(error) 2025-12-04T09:22:55.5216487Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5216600Z Traceback (most recent call last): 2025-12-04T09:22:55.5217135Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5217255Z getattr(self, test_name)() 2025-12-04T09:22:55.5217780Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5217869Z fn() 2025-12-04T09:22:55.5218358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5218464Z method(*args, **kwargs) 2025-12-04T09:22:55.5218953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5219062Z method(*args, **kwargs) 2025-12-04T09:22:55.5219546Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5219640Z with policy(): 2025-12-04T09:22:55.5220139Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5220251Z raise RuntimeError(msg) 2025-12-04T09:22:55.5221817Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.5221832Z 2025-12-04T09:22:55.5222045Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5222839Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5222845Z 2025-12-04T09:22:55.5223114Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5223119Z 2025-12-04T09:22:55.5223128Z 2025-12-04T09:22:55.5223396Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5223826Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5224797Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-bd75c8cd2cf8ff9e.xml - 2025-12-04T09:22:55.5224965Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5225949Z FAILED [8.2736s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5226070Z Traceback (most recent call last): 2025-12-04T09:22:55.5226626Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5227109Z getattr(self, test_name)() 2025-12-04T09:22:55.5227734Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5227833Z fn() 2025-12-04T09:22:55.5228338Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5228449Z method(*args, **kwargs) 2025-12-04T09:22:55.5228949Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5229050Z method(*args, **kwargs) 2025-12-04T09:22:55.5229565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5229660Z with policy(): 2025-12-04T09:22:55.5230165Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5230288Z raise RuntimeError(msg) 2025-12-04T09:22:55.5231633Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.5231640Z 2025-12-04T09:22:55.5231865Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5232688Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5232693Z 2025-12-04T09:22:55.5232966Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5233146Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5233320Z ======================= 1 failed, 9 deselected in 8.30s ======================== 2025-12-04T09:22:55.5233434Z Got exit code 1 2025-12-04T09:22:55.5233541Z Retrying single test... 2025-12-04T09:22:55.5234157Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f7fa0727268f1a56.xml 2025-12-04T09:22:55.5234327Z ============================= test session starts ============================== 2025-12-04T09:22:55.5234676Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5234789Z cachedir: .pytest_cache 2025-12-04T09:22:55.5235305Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5235429Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5235537Z configfile: pytest.ini 2025-12-04T09:22:55.5236072Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5236360Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.5237276Z stepcurrent: skipping 3 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5237390Z Running 1 items in this shard 2025-12-04T09:22:55.5237396Z 2025-12-04T09:22:55.5238587Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda I1204 09:18:52.004000 32332 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 32384 2025-12-04T09:22:55.5239085Z I1204 09:18:52.005000 32332 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 32385 2025-12-04T09:22:55.5239693Z I1204 09:18:52.006000 32332 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 32386 2025-12-04T09:22:55.5240234Z I1204 09:18:52.006000 32332 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 32387 2025-12-04T09:22:55.5241437Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5241574Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5243241Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5243416Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5244619Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5244748Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5246405Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5246563Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5247769Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5247897Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5249563Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5249718Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5250921Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/transformer.py:144: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.self_attn.batch_first was not True(use batch_first for better inference performance) 2025-12-04T09:22:55.5251100Z self.encoder = TransformerEncoder( 2025-12-04T09:22:55.5252750Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5252921Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5253365Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5253896Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5254863Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5255412Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5256375Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5256867Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5257772Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5258238Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5259142Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5259601Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5260509Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5260923Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5261833Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5262305Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5263997Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 1. CUDA driver allocated memory was 598671360 and is now 634322944. 2025-12-04T09:22:55.5264350Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5264968Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5266232Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5266578Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5267305Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5268017Z [rank1]:E1204 09:18:58.758000 32385 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5268117Z dist init r=1, world=4 2025-12-04T09:22:55.5268580Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5269238Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5270244Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5270747Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5271728Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5272133Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5273105Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5273599Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5274552Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5275040Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5276004Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5276452Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5277426Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5277916Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5279819Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 493813760 and is now 634322944. 2025-12-04T09:22:55.5280319Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5280910Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5282033Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5282352Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5282990Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5283525Z [rank3]:E1204 09:18:58.884000 32387 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5283929Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5284397Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5285291Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5285737Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5286610Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5286973Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5287822Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5288264Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5289104Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5289547Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5290390Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5290785Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5291643Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5292074Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5293722Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 2. CUDA driver allocated memory was 602865664 and is now 634322944. 2025-12-04T09:22:55.5294051Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5294643Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5295768Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5296146Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5296786Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5297267Z [rank2]:E1204 09:18:58.915000 32386 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5297366Z dist init r=3, world=4 2025-12-04T09:22:55.5297764Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5298229Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5299128Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5299583Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5300471Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5300820Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5301682Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5302110Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5302967Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5303413Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5304259Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5304659Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5305509Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5306002Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5307827Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 0. CUDA driver allocated memory was 714014720 and is now 743374848. 2025-12-04T09:22:55.5308203Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5308858Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5310187Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5310559Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5311270Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5311821Z [rank0]:E1204 09:18:59.051000 32384 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5311926Z dist init r=2, world=4 2025-12-04T09:22:55.5312024Z dist init r=0, world=4 2025-12-04T09:22:55.5313196Z [rank0]:[W1204 09:18:59.382553179 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5313299Z FAILED [8.5354s] [100%] 2025-12-04T09:22:55.5313305Z 2025-12-04T09:22:55.5313463Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5313900Z _ TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda _ 2025-12-04T09:22:55.5314022Z Traceback (most recent call last): 2025-12-04T09:22:55.5314576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5314690Z self._join_processes(fn) 2025-12-04T09:22:55.5315284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5315429Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5316043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5316166Z raise RuntimeError(error) 2025-12-04T09:22:55.5316407Z RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5316526Z Traceback (most recent call last): 2025-12-04T09:22:55.5317078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5317188Z getattr(self, test_name)() 2025-12-04T09:22:55.5317728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5317819Z fn() 2025-12-04T09:22:55.5318323Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5318434Z method(*args, **kwargs) 2025-12-04T09:22:55.5319027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5319129Z method(*args, **kwargs) 2025-12-04T09:22:55.5319747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5319842Z with policy(): 2025-12-04T09:22:55.5320615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5320718Z raise RuntimeError(msg) 2025-12-04T09:22:55.5321979Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 493813760 and is now 634322944. 2025-12-04T09:22:55.5321992Z 2025-12-04T09:22:55.5322250Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5323021Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5323027Z 2025-12-04T09:22:55.5323278Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5323283Z 2025-12-04T09:22:55.5323288Z 2025-12-04T09:22:55.5323491Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5323931Z Process 3 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5324893Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f7fa0727268f1a56.xml - 2025-12-04T09:22:55.5325062Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5326046Z FAILED [8.5354s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda - RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5326163Z Traceback (most recent call last): 2025-12-04T09:22:55.5326716Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5326826Z getattr(self, test_name)() 2025-12-04T09:22:55.5327355Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5327451Z fn() 2025-12-04T09:22:55.5327958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5328060Z method(*args, **kwargs) 2025-12-04T09:22:55.5328569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5328676Z method(*args, **kwargs) 2025-12-04T09:22:55.5329181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5329276Z with policy(): 2025-12-04T09:22:55.5329778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5329939Z raise RuntimeError(msg) 2025-12-04T09:22:55.5331374Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 19456 on device 3. CUDA driver allocated memory was 493813760 and is now 634322944. 2025-12-04T09:22:55.5331381Z 2025-12-04T09:22:55.5331601Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5332522Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5332532Z 2025-12-04T09:22:55.5332793Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5332978Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5333152Z ======================= 1 failed, 9 deselected in 8.56s ======================== 2025-12-04T09:22:55.5333253Z Got exit code 1 2025-12-04T09:22:55.5333995Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.5334396Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.5335025Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f90badc543da9251.xml 2025-12-04T09:22:55.5335261Z ============================= test session starts ============================== 2025-12-04T09:22:55.5335617Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5335725Z cachedir: .pytest_cache 2025-12-04T09:22:55.5336343Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5336467Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5336570Z configfile: pytest.ini 2025-12-04T09:22:55.5337082Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5337291Z collecting ... collected 10 items / 4 deselected / 6 selected 2025-12-04T09:22:55.5337422Z stepcurrent: skipping 4 already run items. 2025-12-04T09:22:55.5337531Z Running 6 items in this shard 2025-12-04T09:22:55.5337540Z 2025-12-04T09:22:55.5338678Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda I1204 09:19:05.634000 32669 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 32721 2025-12-04T09:22:55.5339159Z I1204 09:19:05.635000 32669 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 32722 2025-12-04T09:22:55.5339644Z I1204 09:19:05.636000 32669 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 32723 2025-12-04T09:22:55.5340115Z I1204 09:19:05.636000 32669 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 32724 2025-12-04T09:22:55.5341792Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5341957Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5343796Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5343940Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5345519Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5345668Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5347191Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5347404Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5348005Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5348589Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5349657Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5350168Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5351153Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5351551Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5352517Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5353009Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5353971Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5354454Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5355412Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5355853Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5356824Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5357319Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5359106Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5359477Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5360261Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5361525Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5361845Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5362475Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5362964Z [rank2]:E1204 09:19:12.181000 32723 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5363418Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5363893Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5364778Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5365229Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5366109Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5366467Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5367317Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5367748Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5368599Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5369028Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5369880Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5370277Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5371126Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5371565Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5373211Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.5373540Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5374118Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5375245Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5375564Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5376199Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5376734Z [rank1]:E1204 09:19:12.181000 32722 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5377131Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5377603Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5378489Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5378937Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5379813Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5380162Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5381015Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5381445Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5382297Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5382731Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5383587Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5383979Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5384831Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5385270Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5386903Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5387288Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5388072Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5389340Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5389768Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5390478Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5391027Z [rank3]:E1204 09:19:12.192000 32724 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5391127Z dist init r=2, world=4 2025-12-04T09:22:55.5391232Z dist init r=1, world=4 2025-12-04T09:22:55.5391329Z dist init r=3, world=4 2025-12-04T09:22:55.5391775Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5392307Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5393312Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5393818Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5394799Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5395189Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5396158Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5396654Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5397610Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5398090Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5399050Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5399491Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5400561Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5401007Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5402588Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.5402914Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5403548Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5404682Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5405001Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5405642Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5406122Z [rank0]:E1204 09:19:12.297000 32721 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5406214Z dist init r=0, world=4 2025-12-04T09:22:55.5407255Z [rank0]:[W1204 09:19:12.609098223 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5407346Z FAILED [8.9212s] [ 16%] 2025-12-04T09:22:55.5407351Z 2025-12-04T09:22:55.5407494Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5407878Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda _ 2025-12-04T09:22:55.5407984Z Traceback (most recent call last): 2025-12-04T09:22:55.5408475Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5408572Z self._join_processes(fn) 2025-12-04T09:22:55.5409097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5409227Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5409759Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5409873Z raise RuntimeError(error) 2025-12-04T09:22:55.5410079Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5410185Z Traceback (most recent call last): 2025-12-04T09:22:55.5410673Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5410769Z getattr(self, test_name)() 2025-12-04T09:22:55.5411246Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5411325Z fn() 2025-12-04T09:22:55.5411769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5411925Z method(*args, **kwargs) 2025-12-04T09:22:55.5412370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5412462Z method(*args, **kwargs) 2025-12-04T09:22:55.5412915Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5413000Z with policy(): 2025-12-04T09:22:55.5413457Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5413556Z raise RuntimeError(msg) 2025-12-04T09:22:55.5414745Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5414816Z 2025-12-04T09:22:55.5415015Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5415740Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5415746Z 2025-12-04T09:22:55.5415988Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5415993Z 2025-12-04T09:22:55.5416135Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5416239Z Traceback (most recent call last): 2025-12-04T09:22:55.5416737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5416835Z getattr(self, test_name)() 2025-12-04T09:22:55.5417315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5417403Z fn() 2025-12-04T09:22:55.5417848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5417952Z method(*args, **kwargs) 2025-12-04T09:22:55.5418397Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5418486Z method(*args, **kwargs) 2025-12-04T09:22:55.5418936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5419020Z with policy(): 2025-12-04T09:22:55.5419472Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5419565Z raise RuntimeError(msg) 2025-12-04T09:22:55.5420749Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5420763Z 2025-12-04T09:22:55.5420948Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5421670Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5421675Z 2025-12-04T09:22:55.5421913Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5421917Z 2025-12-04T09:22:55.5421921Z 2025-12-04T09:22:55.5422114Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5422359Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5423120Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f90badc543da9251.xml - 2025-12-04T09:22:55.5423269Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5424509Z FAILED [8.9212s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5424652Z Traceback (most recent call last): 2025-12-04T09:22:55.5425211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5425321Z getattr(self, test_name)() 2025-12-04T09:22:55.5425851Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5426041Z fn() 2025-12-04T09:22:55.5426549Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5426651Z method(*args, **kwargs) 2025-12-04T09:22:55.5427163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5427346Z method(*args, **kwargs) 2025-12-04T09:22:55.5427865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5427958Z with policy(): 2025-12-04T09:22:55.5428465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5428578Z raise RuntimeError(msg) 2025-12-04T09:22:55.5429915Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5429926Z 2025-12-04T09:22:55.5430149Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5430972Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5430977Z 2025-12-04T09:22:55.5431248Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5431254Z 2025-12-04T09:22:55.5431417Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5431535Z Traceback (most recent call last): 2025-12-04T09:22:55.5432088Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5432198Z getattr(self, test_name)() 2025-12-04T09:22:55.5432740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5432839Z fn() 2025-12-04T09:22:55.5433339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5433450Z method(*args, **kwargs) 2025-12-04T09:22:55.5433953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5434053Z method(*args, **kwargs) 2025-12-04T09:22:55.5434562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5434659Z with policy(): 2025-12-04T09:22:55.5435162Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5435280Z raise RuntimeError(msg) 2025-12-04T09:22:55.5436704Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5436711Z 2025-12-04T09:22:55.5436933Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5437749Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5437755Z 2025-12-04T09:22:55.5438021Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5438200Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5438377Z ======================= 1 failed, 4 deselected in 8.94s ======================== 2025-12-04T09:22:55.5438538Z Got exit code 1 2025-12-04T09:22:55.5438646Z Retrying single test... 2025-12-04T09:22:55.5439373Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-31be29224184e218.xml 2025-12-04T09:22:55.5439537Z ============================= test session starts ============================== 2025-12-04T09:22:55.5439864Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5439976Z cachedir: .pytest_cache 2025-12-04T09:22:55.5440455Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5440568Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5440673Z configfile: pytest.ini 2025-12-04T09:22:55.5441173Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5441376Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.5442229Z stepcurrent: skipping 4 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5442335Z Running 1 items in this shard 2025-12-04T09:22:55.5442340Z 2025-12-04T09:22:55.5443573Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda I1204 09:19:18.994000 33006 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 33058 2025-12-04T09:22:55.5444015Z I1204 09:19:18.995000 33006 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 33059 2025-12-04T09:22:55.5444460Z I1204 09:19:18.996000 33006 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 33060 2025-12-04T09:22:55.5444897Z I1204 09:19:18.996000 33006 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 33061 2025-12-04T09:22:55.5446430Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5446590Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5448100Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5448308Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5449821Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5449973Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5451469Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5451670Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5452080Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5452550Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5453443Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5453891Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5454779Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5455134Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5455991Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5456423Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5457271Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5457705Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5458557Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5458960Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5459818Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5460258Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5461887Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5462225Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5462807Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5463931Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5464266Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5464960Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5465448Z [rank1]:E1204 09:19:25.521000 33059 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5465849Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5466317Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5467213Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5467901Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5468895Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5469290Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5470252Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5470734Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5471690Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5472191Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5473146Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5473597Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5474558Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5475113Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5476905Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 602865664 and is now 630128640. 2025-12-04T09:22:55.5477271Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5477925Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5479195Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5479781Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5480554Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5481042Z [rank2]:E1204 09:19:25.560000 33060 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5481442Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5481913Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5482808Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5483257Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5484140Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5484492Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5485345Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5485781Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5486630Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5487071Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5488126Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5488718Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5489712Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5490195Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5491930Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 498008064 and is now 630128640. 2025-12-04T09:22:55.5492291Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5492932Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5494218Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5494579Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5495271Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5495805Z [rank3]:E1204 09:19:25.561000 33061 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5495907Z dist init r=1, world=4 2025-12-04T09:22:55.5496003Z dist init r=2, world=4 2025-12-04T09:22:55.5496113Z dist init r=3, world=4 2025-12-04T09:22:55.5496550Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5497068Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5498035Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5498528Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5499494Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5499885Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5501059Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5501491Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5502352Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5502782Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5503680Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5504086Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5504943Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5505387Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5506979Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.5507415Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5508216Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5509482Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5509853Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5510573Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5511127Z [rank0]:E1204 09:19:25.706000 33058 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5511230Z dist init r=0, world=4 2025-12-04T09:22:55.5512388Z [rank0]:[W1204 09:19:26.054136393 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5512491Z FAILED [8.6818s] [100%] 2025-12-04T09:22:55.5512497Z 2025-12-04T09:22:55.5512645Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5513091Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda _ 2025-12-04T09:22:55.5513218Z Traceback (most recent call last): 2025-12-04T09:22:55.5513764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5513885Z self._join_processes(fn) 2025-12-04T09:22:55.5514465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5514609Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5515212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5515326Z raise RuntimeError(error) 2025-12-04T09:22:55.5515563Z RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5515684Z Traceback (most recent call last): 2025-12-04T09:22:55.5516232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5516420Z getattr(self, test_name)() 2025-12-04T09:22:55.5516953Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5517044Z fn() 2025-12-04T09:22:55.5517545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5517647Z method(*args, **kwargs) 2025-12-04T09:22:55.5518154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5518255Z method(*args, **kwargs) 2025-12-04T09:22:55.5518758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5518857Z with policy(): 2025-12-04T09:22:55.5519362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5519536Z raise RuntimeError(msg) 2025-12-04T09:22:55.5520882Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 498008064 and is now 630128640. 2025-12-04T09:22:55.5520888Z 2025-12-04T09:22:55.5521088Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5521815Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5521821Z 2025-12-04T09:22:55.5522054Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5522059Z 2025-12-04T09:22:55.5522069Z 2025-12-04T09:22:55.5522270Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5522502Z Process 3 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5523215Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-31be29224184e218.xml - 2025-12-04T09:22:55.5523363Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5524617Z FAILED [8.6818s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda - RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5524749Z Traceback (most recent call last): 2025-12-04T09:22:55.5525301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5525422Z getattr(self, test_name)() 2025-12-04T09:22:55.5525966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5526054Z fn() 2025-12-04T09:22:55.5526565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5526668Z method(*args, **kwargs) 2025-12-04T09:22:55.5527179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5527280Z method(*args, **kwargs) 2025-12-04T09:22:55.5527785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5527889Z with policy(): 2025-12-04T09:22:55.5528395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5528501Z raise RuntimeError(msg) 2025-12-04T09:22:55.5529955Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 498008064 and is now 630128640. 2025-12-04T09:22:55.5529963Z 2025-12-04T09:22:55.5530180Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5531006Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5531012Z 2025-12-04T09:22:55.5531276Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5531459Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5531635Z ======================= 1 failed, 9 deselected in 8.70s ======================== 2025-12-04T09:22:55.5531803Z Got exit code 1 2025-12-04T09:22:55.5531920Z Retrying single test... 2025-12-04T09:22:55.5532542Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1c567b53e42c8343.xml 2025-12-04T09:22:55.5532701Z ============================= test session starts ============================== 2025-12-04T09:22:55.5533054Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5533161Z cachedir: .pytest_cache 2025-12-04T09:22:55.5533680Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5533799Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5533906Z configfile: pytest.ini 2025-12-04T09:22:55.5534453Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5534672Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.5535574Z stepcurrent: skipping 4 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5535698Z Running 1 items in this shard 2025-12-04T09:22:55.5535703Z 2025-12-04T09:22:55.5536946Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda I1204 09:19:32.394000 33343 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 33395 2025-12-04T09:22:55.5537394Z I1204 09:19:32.395000 33343 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 33396 2025-12-04T09:22:55.5537826Z I1204 09:19:32.396000 33343 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 33397 2025-12-04T09:22:55.5538270Z I1204 09:19:32.396000 33343 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 33398 2025-12-04T09:22:55.5539796Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5539952Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5541461Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5541661Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5543170Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5543313Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5544820Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5545021Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5545433Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5545902Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5546793Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5547307Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5548428Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5548837Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5549797Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5550297Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5551256Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5551740Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5552710Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5553153Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5554124Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5554611Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5556477Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5556847Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5557515Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5558787Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5559147Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5560146Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5560625Z [rank1]:E1204 09:19:38.933000 33396 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5561035Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5561506Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5562391Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5562853Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5563729Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5564090Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5564943Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5565385Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5566235Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5566670Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5567525Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5567921Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5568781Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5569673Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5571282Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.5571605Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5572197Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5573324Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5573700Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5574341Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5574822Z [rank2]:E1204 09:19:38.972000 33397 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5575232Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5575700Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5576593Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5577053Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5577920Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5578283Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5579131Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5579578Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5580427Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5580853Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5581713Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5582103Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5583019Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5583455Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5585046Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 602865664 and is now 630128640. 2025-12-04T09:22:55.5585370Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5585958Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5587135Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5587519Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5588399Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5588941Z [rank3]:E1204 09:19:38.978000 33398 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5589056Z dist init r=1, world=4 2025-12-04T09:22:55.5589511Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5590038Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5591043Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5591545Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5592540Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5592938Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5593908Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5594394Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5595353Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5595848Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5596868Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5597328Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5598290Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5598785Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5600619Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.5600992Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5601579Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5602703Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5603032Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5603665Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5604157Z [rank0]:E1204 09:19:39.084000 33395 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5604249Z dist init r=3, world=4 2025-12-04T09:22:55.5604339Z dist init r=2, world=4 2025-12-04T09:22:55.5604433Z dist init r=0, world=4 2025-12-04T09:22:55.5605452Z [rank0]:[W1204 09:19:39.400703315 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5605545Z FAILED [8.3800s] [100%] 2025-12-04T09:22:55.5605550Z 2025-12-04T09:22:55.5605681Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5606070Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda _ 2025-12-04T09:22:55.5606194Z Traceback (most recent call last): 2025-12-04T09:22:55.5606679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5606776Z self._join_processes(fn) 2025-12-04T09:22:55.5607298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5607421Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5607966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5608066Z raise RuntimeError(error) 2025-12-04T09:22:55.5608271Z RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5608386Z Traceback (most recent call last): 2025-12-04T09:22:55.5608871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5609022Z getattr(self, test_name)() 2025-12-04T09:22:55.5609501Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5609581Z fn() 2025-12-04T09:22:55.5610037Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5610132Z method(*args, **kwargs) 2025-12-04T09:22:55.5610577Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5610843Z method(*args, **kwargs) 2025-12-04T09:22:55.5611313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5611410Z with policy(): 2025-12-04T09:22:55.5611889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5612054Z raise RuntimeError(msg) 2025-12-04T09:22:55.5613319Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 602865664 and is now 630128640. 2025-12-04T09:22:55.5613325Z 2025-12-04T09:22:55.5613528Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5614303Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5614309Z 2025-12-04T09:22:55.5614556Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5614562Z 2025-12-04T09:22:55.5614566Z 2025-12-04T09:22:55.5614778Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5615028Z Process 3 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5615777Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1c567b53e42c8343.xml - 2025-12-04T09:22:55.5615943Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5616857Z FAILED [8.3800s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda - RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5616969Z Traceback (most recent call last): 2025-12-04T09:22:55.5617497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5617598Z getattr(self, test_name)() 2025-12-04T09:22:55.5618117Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5618201Z fn() 2025-12-04T09:22:55.5618674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5618778Z method(*args, **kwargs) 2025-12-04T09:22:55.5619483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5619613Z method(*args, **kwargs) 2025-12-04T09:22:55.5620095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5620222Z with policy(): 2025-12-04T09:22:55.5620758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5620869Z raise RuntimeError(msg) 2025-12-04T09:22:55.5622365Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 602865664 and is now 630128640. 2025-12-04T09:22:55.5622381Z 2025-12-04T09:22:55.5622653Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5623513Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5623519Z 2025-12-04T09:22:55.5623967Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5624145Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5624325Z ======================= 1 failed, 9 deselected in 8.40s ======================== 2025-12-04T09:22:55.5624713Z Got exit code 1 2025-12-04T09:22:55.5625525Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda 2025-12-04T09:22:55.5625934Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.5626623Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-3db6e54b58890527.xml 2025-12-04T09:22:55.5626783Z ============================= test session starts ============================== 2025-12-04T09:22:55.5627203Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5627394Z cachedir: .pytest_cache 2025-12-04T09:22:55.5627989Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5628116Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5628224Z configfile: pytest.ini 2025-12-04T09:22:55.5628838Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5629049Z collecting ... collected 10 items / 5 deselected / 5 selected 2025-12-04T09:22:55.5629257Z stepcurrent: skipping 5 already run items. 2025-12-04T09:22:55.5629369Z Running 5 items in this shard 2025-12-04T09:22:55.5629375Z 2025-12-04T09:22:55.5630628Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda I1204 09:19:45.784000 33680 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 33732 2025-12-04T09:22:55.5631202Z I1204 09:19:45.785000 33680 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 33733 2025-12-04T09:22:55.5631772Z I1204 09:19:45.786000 33680 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 33734 2025-12-04T09:22:55.5632332Z I1204 09:19:45.786000 33680 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 33735 2025-12-04T09:22:55.5634194Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5634360Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5636382Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5636552Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5638468Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5638633Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5639262Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5639927Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5641029Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5641572Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5642572Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5643011Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5643986Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5644515Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5645479Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5645992Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5646972Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5647398Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5648447Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5648910Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5650803Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 598671360 and is now 632225792. 2025-12-04T09:22:55.5651144Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5651899Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5653230Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5653629Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5654352Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5654881Z [rank1]:E1204 09:19:52.235000 33733 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5656752Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5656906Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5657011Z dist init r=1, world=4 2025-12-04T09:22:55.5657496Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5657998Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5659016Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5659498Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5660493Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5660866Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5662024Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5662500Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5663431Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5663905Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5664833Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5665271Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5666331Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5666817Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5668816Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.5669179Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5669842Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5671203Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5671579Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5672296Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5672922Z [rank2]:E1204 09:19:52.398000 33734 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5673369Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5673905Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5674916Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5675420Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5676417Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5676813Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5677780Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5678268Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5679224Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5679824Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5680820Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5681282Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5682132Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5682573Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5684152Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5684526Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5685112Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5686303Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5686636Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5687268Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5687759Z [rank3]:E1204 09:19:52.410000 33735 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5687850Z dist init r=2, world=4 2025-12-04T09:22:55.5688245Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5688722Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5689601Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5690049Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5690927Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5691281Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5692133Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5692560Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5693413Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5693951Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5694805Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5695202Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5696058Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5696502Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5698090Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.5698475Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5699056Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5700189Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5700514Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5701161Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5706698Z [rank0]:E1204 09:19:52.512000 33732 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5706832Z dist init r=3, world=4 2025-12-04T09:22:55.5706927Z dist init r=0, world=4 2025-12-04T09:22:55.5708266Z [rank0]:[W1204 09:19:52.855167373 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5708370Z FAILED [8.6268s] [ 20%] 2025-12-04T09:22:55.5708378Z 2025-12-04T09:22:55.5708536Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5708993Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda _ 2025-12-04T09:22:55.5709116Z Traceback (most recent call last): 2025-12-04T09:22:55.5709674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5709785Z self._join_processes(fn) 2025-12-04T09:22:55.5710373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5710521Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5711123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5711247Z raise RuntimeError(error) 2025-12-04T09:22:55.5711484Z RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5711605Z Traceback (most recent call last): 2025-12-04T09:22:55.5712330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5712444Z getattr(self, test_name)() 2025-12-04T09:22:55.5712978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5713078Z fn() 2025-12-04T09:22:55.5713583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5713699Z method(*args, **kwargs) 2025-12-04T09:22:55.5714199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5714302Z method(*args, **kwargs) 2025-12-04T09:22:55.5714813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5714971Z with policy(): 2025-12-04T09:22:55.5715484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5715598Z raise RuntimeError(msg) 2025-12-04T09:22:55.5716940Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5716947Z 2025-12-04T09:22:55.5717170Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5717994Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5718000Z 2025-12-04T09:22:55.5718278Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5718293Z 2025-12-04T09:22:55.5718297Z 2025-12-04T09:22:55.5718517Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5718782Z Process 3 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5719592Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-3db6e54b58890527.xml - 2025-12-04T09:22:55.5719759Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5720827Z FAILED [8.6268s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda - RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5720946Z Traceback (most recent call last): 2025-12-04T09:22:55.5721479Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5721596Z getattr(self, test_name)() 2025-12-04T09:22:55.5722116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5722212Z fn() 2025-12-04T09:22:55.5722703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5722802Z method(*args, **kwargs) 2025-12-04T09:22:55.5723298Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5723393Z method(*args, **kwargs) 2025-12-04T09:22:55.5724241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5724353Z with policy(): 2025-12-04T09:22:55.5724986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5725107Z raise RuntimeError(msg) 2025-12-04T09:22:55.5726452Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5726458Z 2025-12-04T09:22:55.5726673Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5727501Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5727507Z 2025-12-04T09:22:55.5727772Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5727958Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5728213Z ======================= 1 failed, 5 deselected in 8.65s ======================== 2025-12-04T09:22:55.5728310Z Got exit code 1 2025-12-04T09:22:55.5728422Z Retrying single test... 2025-12-04T09:22:55.5729044Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-5f8a5af0d826239e.xml 2025-12-04T09:22:55.5729214Z ============================= test session starts ============================== 2025-12-04T09:22:55.5729558Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5729664Z cachedir: .pytest_cache 2025-12-04T09:22:55.5730180Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5730301Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5730404Z configfile: pytest.ini 2025-12-04T09:22:55.5730950Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5731154Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.5732057Z stepcurrent: skipping 5 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5732164Z Running 1 items in this shard 2025-12-04T09:22:55.5732169Z 2025-12-04T09:22:55.5733338Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda I1204 09:19:59.324000 34017 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 34069 2025-12-04T09:22:55.5733843Z I1204 09:19:59.325000 34017 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 34070 2025-12-04T09:22:55.5734339Z I1204 09:19:59.326000 34017 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 34071 2025-12-04T09:22:55.5734831Z I1204 09:19:59.327000 34017 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 34072 2025-12-04T09:22:55.5736760Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5736912Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5738474Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5738631Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5740132Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5740274Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5741785Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5741978Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5742394Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5742864Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5743763Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5744213Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5745088Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5745447Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5746300Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5746738Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5747843Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5748346Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5749304Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5749752Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5750730Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5751220Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5753077Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5753446Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5754109Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5755374Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5755807Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5756526Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5757067Z [rank2]:E1204 09:20:05.895000 34071 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5757526Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5758054Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5759058Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5759673Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5760698Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5761056Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5761911Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5762344Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5763197Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5763634Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5764571Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5765029Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5765950Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5766386Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5767972Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.5768293Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5768886Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5770065Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5770397Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5771031Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5771511Z [rank1]:E1204 09:20:05.911000 34070 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5771913Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5772383Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5773274Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5773718Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5774593Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5774956Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5775808Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5776245Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5777091Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5777528Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5778372Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5778831Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5779687Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5780124Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5781713Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 485425152 and is now 630128640. 2025-12-04T09:22:55.5782036Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5782674Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5783793Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5784123Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5784755Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5785234Z [rank3]:E1204 09:20:05.923000 34072 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5785339Z dist init r=2, world=4 2025-12-04T09:22:55.5785425Z dist init r=3, world=4 2025-12-04T09:22:55.5785512Z dist init r=1, world=4 2025-12-04T09:22:55.5785915Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5786383Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5787325Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5787962Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5788959Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5789356Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5790309Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5790795Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5791747Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5792307Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5793264Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5793705Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5794665Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5795147Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5796940Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.5797357Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5798015Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5799284Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5799657Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5800441Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5800923Z [rank0]:E1204 09:20:06.098000 34069 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5801018Z dist init r=0, world=4 2025-12-04T09:22:55.5802033Z [rank0]:[W1204 09:20:06.400611119 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5802128Z FAILED [9.2632s] [100%] 2025-12-04T09:22:55.5802133Z 2025-12-04T09:22:55.5802265Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5802656Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda _ 2025-12-04T09:22:55.5802768Z Traceback (most recent call last): 2025-12-04T09:22:55.5803253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5803357Z self._join_processes(fn) 2025-12-04T09:22:55.5803871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5803995Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5804537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5804638Z raise RuntimeError(error) 2025-12-04T09:22:55.5804841Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.5804956Z Traceback (most recent call last): 2025-12-04T09:22:55.5805482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5805588Z getattr(self, test_name)() 2025-12-04T09:22:55.5806061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5806142Z fn() 2025-12-04T09:22:55.5806594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5806685Z method(*args, **kwargs) 2025-12-04T09:22:55.5807126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5807222Z method(*args, **kwargs) 2025-12-04T09:22:55.5807664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5807804Z with policy(): 2025-12-04T09:22:55.5808257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5808352Z raise RuntimeError(msg) 2025-12-04T09:22:55.5809545Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.5809550Z 2025-12-04T09:22:55.5809738Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5810467Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5810472Z 2025-12-04T09:22:55.5810714Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5810723Z 2025-12-04T09:22:55.5810872Z Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5810979Z Traceback (most recent call last): 2025-12-04T09:22:55.5811459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5811563Z getattr(self, test_name)() 2025-12-04T09:22:55.5812033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5812110Z fn() 2025-12-04T09:22:55.5812560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5812647Z method(*args, **kwargs) 2025-12-04T09:22:55.5813099Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5813191Z method(*args, **kwargs) 2025-12-04T09:22:55.5813639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5813733Z with policy(): 2025-12-04T09:22:55.5814179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5814271Z raise RuntimeError(msg) 2025-12-04T09:22:55.5815467Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5815472Z 2025-12-04T09:22:55.5815660Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5816436Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5816446Z 2025-12-04T09:22:55.5816679Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5816684Z 2025-12-04T09:22:55.5816831Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5816936Z Traceback (most recent call last): 2025-12-04T09:22:55.5817419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5817524Z getattr(self, test_name)() 2025-12-04T09:22:55.5817993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5818077Z fn() 2025-12-04T09:22:55.5818530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5818681Z method(*args, **kwargs) 2025-12-04T09:22:55.5819132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5819220Z method(*args, **kwargs) 2025-12-04T09:22:55.5819662Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5819754Z with policy(): 2025-12-04T09:22:55.5820199Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5820297Z raise RuntimeError(msg) 2025-12-04T09:22:55.5821473Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 485425152 and is now 630128640. 2025-12-04T09:22:55.5821481Z 2025-12-04T09:22:55.5821670Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5822399Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5822404Z 2025-12-04T09:22:55.5822631Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5822636Z 2025-12-04T09:22:55.5822640Z 2025-12-04T09:22:55.5822839Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5823068Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5823946Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-5f8a5af0d826239e.xml - 2025-12-04T09:22:55.5824282Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5825263Z FAILED [9.2632s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.5825393Z Traceback (most recent call last): 2025-12-04T09:22:55.5825936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5826045Z getattr(self, test_name)() 2025-12-04T09:22:55.5826589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5826676Z fn() 2025-12-04T09:22:55.5827190Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5827368Z method(*args, **kwargs) 2025-12-04T09:22:55.5827975Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5828088Z method(*args, **kwargs) 2025-12-04T09:22:55.5828588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5828691Z with policy(): 2025-12-04T09:22:55.5829194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5829298Z raise RuntimeError(msg) 2025-12-04T09:22:55.5830644Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.5830650Z 2025-12-04T09:22:55.5830861Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5831767Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5831774Z 2025-12-04T09:22:55.5832038Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5832043Z 2025-12-04T09:22:55.5832204Z Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.5832330Z Traceback (most recent call last): 2025-12-04T09:22:55.5832872Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5832991Z getattr(self, test_name)() 2025-12-04T09:22:55.5833527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5833615Z fn() 2025-12-04T09:22:55.5834126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5834236Z method(*args, **kwargs) 2025-12-04T09:22:55.5834740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5834847Z method(*args, **kwargs) 2025-12-04T09:22:55.5835345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5835446Z with policy(): 2025-12-04T09:22:55.5835952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5836059Z raise RuntimeError(msg) 2025-12-04T09:22:55.5837402Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5837415Z 2025-12-04T09:22:55.5837626Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5838449Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5838454Z 2025-12-04T09:22:55.5838716Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5838721Z 2025-12-04T09:22:55.5838890Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5839005Z Traceback (most recent call last): 2025-12-04T09:22:55.5839628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5839732Z getattr(self, test_name)() 2025-12-04T09:22:55.5840205Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5840337Z fn() 2025-12-04T09:22:55.5840790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5840878Z method(*args, **kwargs) 2025-12-04T09:22:55.5841330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5841420Z method(*args, **kwargs) 2025-12-04T09:22:55.5841863Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5841956Z with policy(): 2025-12-04T09:22:55.5842401Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5842493Z raise RuntimeError(msg) 2025-12-04T09:22:55.5843685Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 485425152 and is now 630128640. 2025-12-04T09:22:55.5843740Z 2025-12-04T09:22:55.5843927Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5844652Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5844656Z 2025-12-04T09:22:55.5844884Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5845053Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5845207Z ======================= 1 failed, 9 deselected in 9.29s ======================== 2025-12-04T09:22:55.5845295Z Got exit code 1 2025-12-04T09:22:55.5845399Z Retrying single test... 2025-12-04T09:22:55.5845951Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1a3a2c29e874821d.xml 2025-12-04T09:22:55.5846095Z ============================= test session starts ============================== 2025-12-04T09:22:55.5846411Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5846506Z cachedir: .pytest_cache 2025-12-04T09:22:55.5846967Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5847074Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5847166Z configfile: pytest.ini 2025-12-04T09:22:55.5847649Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5847833Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.5848639Z stepcurrent: skipping 5 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5848744Z Running 1 items in this shard 2025-12-04T09:22:55.5848749Z 2025-12-04T09:22:55.5849972Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda I1204 09:20:12.784000 34354 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 34406 2025-12-04T09:22:55.5850447Z I1204 09:20:12.785000 34354 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 34407 2025-12-04T09:22:55.5850908Z I1204 09:20:12.786000 34354 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 34408 2025-12-04T09:22:55.5851372Z I1204 09:20:12.787000 34354 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 34409 2025-12-04T09:22:55.5853040Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5853196Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5854801Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5855002Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5856797Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5856950Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5858604Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5858761Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5859213Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5859729Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5860698Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5861192Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5862142Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5862538Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5863468Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5863941Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5864866Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5865335Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5866318Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5866752Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5867920Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5868410Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5870204Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5870647Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5871302Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5872578Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5872940Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5873662Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5874208Z [rank1]:E1204 09:20:19.307000 34407 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5874665Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5875197Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5876194Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5876703Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5877698Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5878103Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5879060Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5879662Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5880565Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5881076Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5881982Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5882401Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5883310Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5883770Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5885499Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.5885844Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5886458Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5887650Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5887997Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5888666Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5889182Z [rank2]:E1204 09:20:19.314000 34408 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5889599Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5890098Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5891035Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5891508Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5892437Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5892807Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5893711Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5894217Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5895119Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5895573Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5896470Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5896890Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5897793Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5898303Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5899983Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 495910912 and is now 630128640. 2025-12-04T09:22:55.5900326Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5900939Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5902143Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5902483Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5903147Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5903658Z [rank3]:E1204 09:20:19.319000 34409 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.5903751Z dist init r=1, world=4 2025-12-04T09:22:55.5903841Z dist init r=2, world=4 2025-12-04T09:22:55.5904271Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5904768Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5905711Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5906182Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5907112Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5907543Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5908800Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5909291Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5910247Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5910736Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5911690Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5912188Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5913146Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5913630Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5915418Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.5915782Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5916523Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5917871Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5918238Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5918948Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5919494Z [rank0]:E1204 09:20:19.469000 34406 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.5919597Z dist init r=3, world=4 2025-12-04T09:22:55.5919691Z dist init r=0, world=4 2025-12-04T09:22:55.5920936Z [rank0]:[W1204 09:20:19.800218800 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.5921030Z FAILED [8.9296s] [100%] 2025-12-04T09:22:55.5921035Z 2025-12-04T09:22:55.5921175Z =================================== FAILURES =================================== 2025-12-04T09:22:55.5921602Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda _ 2025-12-04T09:22:55.5921716Z Traceback (most recent call last): 2025-12-04T09:22:55.5922696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.5922807Z self._join_processes(fn) 2025-12-04T09:22:55.5923370Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.5923511Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.5924295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.5924410Z raise RuntimeError(error) 2025-12-04T09:22:55.5924821Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.5924936Z Traceback (most recent call last): 2025-12-04T09:22:55.5925480Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5925689Z getattr(self, test_name)() 2025-12-04T09:22:55.5926224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5926315Z fn() 2025-12-04T09:22:55.5926815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5926922Z method(*args, **kwargs) 2025-12-04T09:22:55.5927424Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5927524Z method(*args, **kwargs) 2025-12-04T09:22:55.5928026Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5928119Z with policy(): 2025-12-04T09:22:55.5928622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5928735Z raise RuntimeError(msg) 2025-12-04T09:22:55.5930070Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5930077Z 2025-12-04T09:22:55.5930294Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5931107Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5931113Z 2025-12-04T09:22:55.5931379Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5931384Z 2025-12-04T09:22:55.5931544Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5931661Z Traceback (most recent call last): 2025-12-04T09:22:55.5932217Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5932322Z getattr(self, test_name)() 2025-12-04T09:22:55.5932850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5932941Z fn() 2025-12-04T09:22:55.5933443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5933551Z method(*args, **kwargs) 2025-12-04T09:22:55.5934049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5934147Z method(*args, **kwargs) 2025-12-04T09:22:55.5934649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5934749Z with policy(): 2025-12-04T09:22:55.5935325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5935437Z raise RuntimeError(msg) 2025-12-04T09:22:55.5936944Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 495910912 and is now 630128640. 2025-12-04T09:22:55.5936949Z 2025-12-04T09:22:55.5937153Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5937914Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5937919Z 2025-12-04T09:22:55.5938172Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5938233Z 2025-12-04T09:22:55.5938241Z 2025-12-04T09:22:55.5938446Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.5938692Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.5939628Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1a3a2c29e874821d.xml - 2025-12-04T09:22:55.5939790Z =========================== short test summary info ============================ 2025-12-04T09:22:55.5940738Z FAILED [8.9296s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.5940851Z Traceback (most recent call last): 2025-12-04T09:22:55.5941384Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5941502Z getattr(self, test_name)() 2025-12-04T09:22:55.5942019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5942109Z fn() 2025-12-04T09:22:55.5942599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5942698Z method(*args, **kwargs) 2025-12-04T09:22:55.5943187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5943284Z method(*args, **kwargs) 2025-12-04T09:22:55.5943768Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5943862Z with policy(): 2025-12-04T09:22:55.5944352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5944466Z raise RuntimeError(msg) 2025-12-04T09:22:55.5945768Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5945773Z 2025-12-04T09:22:55.5945984Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5946774Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5946780Z 2025-12-04T09:22:55.5947031Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5947036Z 2025-12-04T09:22:55.5947203Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.5947442Z Traceback (most recent call last): 2025-12-04T09:22:55.5948154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5948261Z getattr(self, test_name)() 2025-12-04T09:22:55.5948789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5948887Z fn() 2025-12-04T09:22:55.5949387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5949487Z method(*args, **kwargs) 2025-12-04T09:22:55.5949997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5950097Z method(*args, **kwargs) 2025-12-04T09:22:55.5950600Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5950752Z with policy(): 2025-12-04T09:22:55.5951253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5951364Z raise RuntimeError(msg) 2025-12-04T09:22:55.5952706Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 495910912 and is now 630128640. 2025-12-04T09:22:55.5952712Z 2025-12-04T09:22:55.5952925Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5953738Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5953748Z 2025-12-04T09:22:55.5954012Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5954195Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.5954370Z ======================= 1 failed, 9 deselected in 8.95s ======================== 2025-12-04T09:22:55.5954471Z Got exit code 1 2025-12-04T09:22:55.5955207Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda 2025-12-04T09:22:55.5955607Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.5956227Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-097bd188165cd189.xml 2025-12-04T09:22:55.5956385Z ============================= test session starts ============================== 2025-12-04T09:22:55.5956739Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.5956846Z cachedir: .pytest_cache 2025-12-04T09:22:55.5957358Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.5957484Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.5957588Z configfile: pytest.ini 2025-12-04T09:22:55.5958117Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.5958330Z collecting ... collected 10 items / 6 deselected / 4 selected 2025-12-04T09:22:55.5958465Z stepcurrent: skipping 6 already run items. 2025-12-04T09:22:55.5958577Z Running 4 items in this shard 2025-12-04T09:22:55.5958583Z 2025-12-04T09:22:55.5960038Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda I1204 09:20:26.154000 34691 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 34743 2025-12-04T09:22:55.5960478Z I1204 09:20:26.155000 34691 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 34744 2025-12-04T09:22:55.5960916Z I1204 09:20:26.156000 34691 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 34745 2025-12-04T09:22:55.5961344Z I1204 09:20:26.157000 34691 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 34746 2025-12-04T09:22:55.5962869Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5963068Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5964588Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5964729Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5966232Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5966384Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5967884Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.5968031Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.5968436Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5968907Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5969787Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5970241Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5971113Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5971462Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5972313Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5972741Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5973655Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5974082Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5974921Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5975318Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5976167Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5976655Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5978229Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.5978559Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5979136Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5980267Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.5980585Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5981216Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5981700Z [rank2]:E1204 09:20:32.723000 34745 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.5982102Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5982582Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5983466Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5983916Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.5984792Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.5985141Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.5986039Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5986474Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5987385Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.5988010Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.5988968Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.5989503Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.5990462Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.5990955Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.5992731Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.5993100Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5993762Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.5995037Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.5995396Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.5996108Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.5996653Z [rank1]:E1204 09:20:32.742000 34744 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.5997108Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.5997640Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.5998634Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.5999141Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6000215Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6000620Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6001474Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6001902Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6002751Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6003179Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6004081Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6004471Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6005316Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6005750Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6007325Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 498008064 and is now 630128640. 2025-12-04T09:22:55.6007652Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6008228Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6009352Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6009669Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6010301Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6010790Z [rank3]:E1204 09:20:32.754000 34746 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.6010878Z dist init r=2, world=4 2025-12-04T09:22:55.6010966Z dist init r=1, world=4 2025-12-04T09:22:55.6011365Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6011830Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6012712Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6013209Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6014088Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6014435Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6015283Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6015711Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6016555Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6017049Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6017892Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6018284Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6019129Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6019571Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6021142Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.6021462Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6022047Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6023384Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6023871Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6024742Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6025285Z [rank0]:E1204 09:20:32.903000 34743 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.6025383Z dist init r=3, world=4 2025-12-04T09:22:55.6025475Z dist init r=0, world=4 2025-12-04T09:22:55.6026718Z [rank0]:[W1204 09:20:33.214084337 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.6026820Z FAILED [9.2007s] [ 25%] 2025-12-04T09:22:55.6026825Z 2025-12-04T09:22:55.6026974Z =================================== FAILURES =================================== 2025-12-04T09:22:55.6027492Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda _ 2025-12-04T09:22:55.6027612Z Traceback (most recent call last): 2025-12-04T09:22:55.6028157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.6028265Z self._join_processes(fn) 2025-12-04T09:22:55.6028844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.6028987Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.6029589Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.6029778Z raise RuntimeError(error) 2025-12-04T09:22:55.6030010Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.6030129Z Traceback (most recent call last): 2025-12-04T09:22:55.6030670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6035041Z getattr(self, test_name)() 2025-12-04T09:22:55.6035599Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6035698Z fn() 2025-12-04T09:22:55.6036210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6036316Z method(*args, **kwargs) 2025-12-04T09:22:55.6036831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6036946Z method(*args, **kwargs) 2025-12-04T09:22:55.6037455Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6037550Z with policy(): 2025-12-04T09:22:55.6038054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6038184Z raise RuntimeError(msg) 2025-12-04T09:22:55.6039717Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6039724Z 2025-12-04T09:22:55.6039924Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6040704Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6040710Z 2025-12-04T09:22:55.6040958Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6040964Z 2025-12-04T09:22:55.6041114Z Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.6041232Z Traceback (most recent call last): 2025-12-04T09:22:55.6041751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6041852Z getattr(self, test_name)() 2025-12-04T09:22:55.6042356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6042437Z fn() 2025-12-04T09:22:55.6042916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6043101Z method(*args, **kwargs) 2025-12-04T09:22:55.6043574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6043674Z method(*args, **kwargs) 2025-12-04T09:22:55.6044146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6044244Z with policy(): 2025-12-04T09:22:55.6044722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6044822Z raise RuntimeError(msg) 2025-12-04T09:22:55.6046083Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6046125Z 2025-12-04T09:22:55.6046324Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6047284Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6047289Z 2025-12-04T09:22:55.6047520Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6047577Z 2025-12-04T09:22:55.6047721Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.6047835Z Traceback (most recent call last): 2025-12-04T09:22:55.6048316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6048420Z getattr(self, test_name)() 2025-12-04T09:22:55.6048890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6048973Z fn() 2025-12-04T09:22:55.6049425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6049515Z method(*args, **kwargs) 2025-12-04T09:22:55.6049962Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6050061Z method(*args, **kwargs) 2025-12-04T09:22:55.6050504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6050595Z with policy(): 2025-12-04T09:22:55.6051042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6051137Z raise RuntimeError(msg) 2025-12-04T09:22:55.6052322Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 498008064 and is now 630128640. 2025-12-04T09:22:55.6052329Z 2025-12-04T09:22:55.6052515Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6053243Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6053250Z 2025-12-04T09:22:55.6053483Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6053488Z 2025-12-04T09:22:55.6053492Z 2025-12-04T09:22:55.6053691Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.6053920Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.6054686Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-097bd188165cd189.xml - 2025-12-04T09:22:55.6054842Z =========================== short test summary info ============================ 2025-12-04T09:22:55.6055708Z FAILED [9.2007s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.6055823Z Traceback (most recent call last): 2025-12-04T09:22:55.6056305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6056401Z getattr(self, test_name)() 2025-12-04T09:22:55.6056882Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6056962Z fn() 2025-12-04T09:22:55.6057439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6057535Z method(*args, **kwargs) 2025-12-04T09:22:55.6057977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6058071Z method(*args, **kwargs) 2025-12-04T09:22:55.6058515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6058636Z with policy(): 2025-12-04T09:22:55.6059090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6059182Z raise RuntimeError(msg) 2025-12-04T09:22:55.6060370Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6060378Z 2025-12-04T09:22:55.6060565Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6061287Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6061292Z 2025-12-04T09:22:55.6061532Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6061536Z 2025-12-04T09:22:55.6061679Z Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.6061787Z Traceback (most recent call last): 2025-12-04T09:22:55.6062269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6062364Z getattr(self, test_name)() 2025-12-04T09:22:55.6062847Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6062923Z fn() 2025-12-04T09:22:55.6063559Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6063654Z method(*args, **kwargs) 2025-12-04T09:22:55.6064127Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6064228Z method(*args, **kwargs) 2025-12-04T09:22:55.6064699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6064785Z with policy(): 2025-12-04T09:22:55.6065265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6065362Z raise RuntimeError(msg) 2025-12-04T09:22:55.6066695Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6066701Z 2025-12-04T09:22:55.6066906Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6067945Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6067954Z 2025-12-04T09:22:55.6068217Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6068222Z 2025-12-04T09:22:55.6068394Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.6068512Z Traceback (most recent call last): 2025-12-04T09:22:55.6069059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6069219Z getattr(self, test_name)() 2025-12-04T09:22:55.6069751Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6069844Z fn() 2025-12-04T09:22:55.6070345Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6070482Z method(*args, **kwargs) 2025-12-04T09:22:55.6070992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6071093Z method(*args, **kwargs) 2025-12-04T09:22:55.6071590Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6071692Z with policy(): 2025-12-04T09:22:55.6072203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6072316Z raise RuntimeError(msg) 2025-12-04T09:22:55.6073642Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 498008064 and is now 630128640. 2025-12-04T09:22:55.6073650Z 2025-12-04T09:22:55.6073873Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6074680Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6074686Z 2025-12-04T09:22:55.6074945Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6075136Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.6075317Z ======================= 1 failed, 6 deselected in 9.22s ======================== 2025-12-04T09:22:55.6075414Z Got exit code 1 2025-12-04T09:22:55.6075522Z Retrying single test... 2025-12-04T09:22:55.6076152Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-efc9fa1f0bd357d6.xml 2025-12-04T09:22:55.6076319Z ============================= test session starts ============================== 2025-12-04T09:22:55.6076667Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.6076772Z cachedir: .pytest_cache 2025-12-04T09:22:55.6077300Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.6077419Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.6077526Z configfile: pytest.ini 2025-12-04T09:22:55.6078128Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.6078335Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.6079236Z stepcurrent: skipping 6 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6079350Z Running 1 items in this shard 2025-12-04T09:22:55.6079355Z 2025-12-04T09:22:55.6080699Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda I1204 09:20:39.644000 35028 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 35080 2025-12-04T09:22:55.6081165Z I1204 09:20:39.645000 35028 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 35081 2025-12-04T09:22:55.6081661Z I1204 09:20:39.646000 35028 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 35082 2025-12-04T09:22:55.6082123Z I1204 09:20:39.647000 35028 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 35083 2025-12-04T09:22:55.6083750Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6083943Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6085552Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6085714Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6087317Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6087470Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6089078Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6089228Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6089667Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6090164Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6091104Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6091577Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6092560Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6092942Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6093843Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6094306Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6095201Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6095696Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6096589Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6097004Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6097946Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6098402Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6100094Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6100437Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6101056Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6102239Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6102590Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6103266Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6103776Z [rank2]:E1204 09:20:46.041000 35082 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.6104204Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6104815Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6105755Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6106201Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6107072Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6107492Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6108608Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6109102Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6110163Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6110654Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6111642Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6112084Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6113057Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6113551Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6115340Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6115706Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6116362Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6117628Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6117999Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6118710Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6119249Z [rank1]:E1204 09:20:46.125000 35081 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.6119358Z dist init r=2, world=4 2025-12-04T09:22:55.6119904Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6120565Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6121449Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6121895Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6122777Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6123124Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6124360Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6125022Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6125993Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6126559Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6127511Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6127966Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6128927Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6129423Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6131205Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 581894144 and is now 630128640. 2025-12-04T09:22:55.6131580Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6132235Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6133507Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6133878Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6134584Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6135204Z [rank3]:E1204 09:20:46.150000 35083 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.6135308Z dist init r=1, world=4 2025-12-04T09:22:55.6135409Z dist init r=3, world=4 2025-12-04T09:22:55.6135871Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6136509Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6137549Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6137994Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6138882Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6139268Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6140116Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6140582Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6141428Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6141866Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6142707Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6143106Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6143956Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6144386Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6145972Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.6146294Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6146886Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6148287Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6148727Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6149440Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6149985Z [rank0]:E1204 09:20:46.294000 35080 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.6150100Z dist init r=0, world=4 2025-12-04T09:22:55.6151251Z [rank0]:[W1204 09:20:46.606044619 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.6151356Z FAILED [8.7369s] [100%] 2025-12-04T09:22:55.6151362Z 2025-12-04T09:22:55.6151506Z =================================== FAILURES =================================== 2025-12-04T09:22:55.6151969Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda _ 2025-12-04T09:22:55.6152092Z Traceback (most recent call last): 2025-12-04T09:22:55.6152636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.6152752Z self._join_processes(fn) 2025-12-04T09:22:55.6153330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.6153507Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.6154120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.6154232Z raise RuntimeError(error) 2025-12-04T09:22:55.6154470Z RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.6154591Z Traceback (most recent call last): 2025-12-04T09:22:55.6155132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6155244Z getattr(self, test_name)() 2025-12-04T09:22:55.6155773Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6155859Z fn() 2025-12-04T09:22:55.6156372Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6156474Z method(*args, **kwargs) 2025-12-04T09:22:55.6156981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6157084Z method(*args, **kwargs) 2025-12-04T09:22:55.6157583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6157684Z with policy(): 2025-12-04T09:22:55.6158191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6158296Z raise RuntimeError(msg) 2025-12-04T09:22:55.6159746Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 581894144 and is now 630128640. 2025-12-04T09:22:55.6159754Z 2025-12-04T09:22:55.6160082Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6160810Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6160815Z 2025-12-04T09:22:55.6161051Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6161106Z 2025-12-04T09:22:55.6161111Z 2025-12-04T09:22:55.6161309Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.6161541Z Process 3 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.6162445Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-efc9fa1f0bd357d6.xml - 2025-12-04T09:22:55.6162611Z =========================== short test summary info ============================ 2025-12-04T09:22:55.6163524Z FAILED [8.7369s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda - RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.6163638Z Traceback (most recent call last): 2025-12-04T09:22:55.6164157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6164287Z getattr(self, test_name)() 2025-12-04T09:22:55.6164798Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6164880Z fn() 2025-12-04T09:22:55.6165356Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6165495Z method(*args, **kwargs) 2025-12-04T09:22:55.6165965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6166067Z method(*args, **kwargs) 2025-12-04T09:22:55.6166538Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6166624Z with policy(): 2025-12-04T09:22:55.6167108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6167210Z raise RuntimeError(msg) 2025-12-04T09:22:55.6168474Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 581894144 and is now 630128640. 2025-12-04T09:22:55.6168482Z 2025-12-04T09:22:55.6168683Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6169449Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6169462Z 2025-12-04T09:22:55.6169709Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6169874Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.6170055Z ======================= 1 failed, 9 deselected in 8.76s ======================== 2025-12-04T09:22:55.6170145Z Got exit code 1 2025-12-04T09:22:55.6170241Z Retrying single test... 2025-12-04T09:22:55.6170824Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-648e0de30361d669.xml 2025-12-04T09:22:55.6170976Z ============================= test session starts ============================== 2025-12-04T09:22:55.6171306Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.6171403Z cachedir: .pytest_cache 2025-12-04T09:22:55.6171884Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.6172003Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.6172099Z configfile: pytest.ini 2025-12-04T09:22:55.6172672Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.6172874Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.6173716Z stepcurrent: skipping 6 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6173829Z Running 1 items in this shard 2025-12-04T09:22:55.6173834Z 2025-12-04T09:22:55.6174936Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda I1204 09:20:53.074000 35365 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 35417 2025-12-04T09:22:55.6175406Z I1204 09:20:53.075000 35365 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 35418 2025-12-04T09:22:55.6175901Z I1204 09:20:53.076000 35365 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 35419 2025-12-04T09:22:55.6176474Z I1204 09:20:53.076000 35365 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 35420 2025-12-04T09:22:55.6178006Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6178183Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6179705Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6179853Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6181368Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6181513Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6183025Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6183168Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6183573Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6184049Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6184932Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6185383Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6186304Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6186657Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6187782Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6188425Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6189389Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6189909Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6190872Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6191318Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6192306Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6192801Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6194588Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6194956Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6195696Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6197127Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6197565Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6198352Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6198925Z [rank2]:E1204 09:20:59.665000 35419 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.6199411Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6200016Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6201245Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6201813Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6202885Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6203289Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6204295Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6204833Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6205895Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6206406Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6207502Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6207936Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6208953Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6209477Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6211625Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 609157120 and is now 630128640. 2025-12-04T09:22:55.6211955Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6212598Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6214058Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6214463Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6215198Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6215708Z [rank1]:E1204 09:20:59.665000 35418 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.6216196Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6216828Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6217836Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6218384Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6219377Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6219824Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6220791Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6221362Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6222338Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6222897Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6224210Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6224706Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6225775Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6226291Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6228340Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6228716Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6229449Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6230860Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6231290Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6232079Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6232762Z [rank3]:E1204 09:20:59.670000 35420 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.6233219Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6233824Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6235567Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6237290Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6238979Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6240623Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6242143Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6243722Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6245299Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6246828Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6248352Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6249919Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6251410Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6252938Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6255268Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.6257520Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6258597Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6260522Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6262234Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6263446Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6264754Z [rank0]:E1204 09:20:59.768000 35417 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.6265484Z dist init r=1, world=4 2025-12-04T09:22:55.6265747Z dist init r=3, world=4 2025-12-04T09:22:55.6266007Z dist init r=2, world=4 2025-12-04T09:22:55.6266250Z dist init r=0, world=4 2025-12-04T09:22:55.6267558Z [rank0]:[W1204 09:21:00.104126161 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.6269098Z FAILED [8.6206s] [100%] 2025-12-04T09:22:55.6269273Z 2025-12-04T09:22:55.6269432Z =================================== FAILURES =================================== 2025-12-04T09:22:55.6270187Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda _ 2025-12-04T09:22:55.6270876Z Traceback (most recent call last): 2025-12-04T09:22:55.6271654Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.6272429Z self._join_processes(fn) 2025-12-04T09:22:55.6273216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.6274113Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.6274978Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.6275820Z raise RuntimeError(error) 2025-12-04T09:22:55.6276259Z RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.6276742Z Traceback (most recent call last): 2025-12-04T09:22:55.6277514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6278292Z getattr(self, test_name)() 2025-12-04T09:22:55.6279036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6279886Z fn() 2025-12-04T09:22:55.6280476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6281179Z method(*args, **kwargs) 2025-12-04T09:22:55.6281837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6282532Z method(*args, **kwargs) 2025-12-04T09:22:55.6283183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6283867Z with policy(): 2025-12-04T09:22:55.6284493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6285194Z raise RuntimeError(msg) 2025-12-04T09:22:55.6286645Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6288021Z 2025-12-04T09:22:55.6288276Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6289369Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6290253Z 2025-12-04T09:22:55.6290509Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6290948Z 2025-12-04T09:22:55.6290953Z 2025-12-04T09:22:55.6291161Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.6291747Z Process 3 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.6292870Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-648e0de30361d669.xml - 2025-12-04T09:22:55.6293892Z =========================== short test summary info ============================ 2025-12-04T09:22:55.6295269Z FAILED [8.6206s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda - RuntimeError: Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.6296441Z Traceback (most recent call last): 2025-12-04T09:22:55.6297203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6298363Z getattr(self, test_name)() 2025-12-04T09:22:55.6299073Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6299814Z fn() 2025-12-04T09:22:55.6300430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6301189Z method(*args, **kwargs) 2025-12-04T09:22:55.6301875Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6302601Z method(*args, **kwargs) 2025-12-04T09:22:55.6303283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6303989Z with policy(): 2025-12-04T09:22:55.6304647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6305376Z raise RuntimeError(msg) 2025-12-04T09:22:55.6306870Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6308553Z 2025-12-04T09:22:55.6308769Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6309933Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6310883Z 2025-12-04T09:22:55.6311145Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6311721Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.6312207Z ======================= 1 failed, 9 deselected in 8.64s ======================== 2025-12-04T09:22:55.6312622Z Got exit code 1 2025-12-04T09:22:55.6313517Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda 2025-12-04T09:22:55.6314775Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.6315920Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-21aecc6f8d181c25.xml 2025-12-04T09:22:55.6316828Z ============================= test session starts ============================== 2025-12-04T09:22:55.6317478Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.6318063Z cachedir: .pytest_cache 2025-12-04T09:22:55.6318837Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.6319903Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.6320201Z configfile: pytest.ini 2025-12-04T09:22:55.6320827Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.6321604Z collecting ... collected 10 items / 7 deselected / 3 selected 2025-12-04T09:22:55.6322028Z stepcurrent: skipping 7 already run items. 2025-12-04T09:22:55.6322350Z Running 3 items in this shard 2025-12-04T09:22:55.6322534Z 2025-12-04T09:22:55.6323720Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda I1204 09:21:06.514000 35702 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 35754 2025-12-04T09:22:55.6325869Z I1204 09:21:06.515000 35702 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 35755 2025-12-04T09:22:55.6327061Z I1204 09:21:06.516000 35702 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 35756 2025-12-04T09:22:55.6328176Z I1204 09:21:06.517000 35702 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 35757 2025-12-04T09:22:55.6330504Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6332563Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6334573Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6336671Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6338585Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6340431Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6342271Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6344043Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6344708Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6345703Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6347180Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6349106Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6350744Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6352267Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6353766Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6355344Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6356927Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6358538Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6360214Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6361611Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6362971Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6364383Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6366524Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6368550Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6369578Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6371407Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6372961Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6374037Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6375277Z [rank2]:E1204 09:21:13.044000 35756 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.6376287Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6377266Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6378795Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6380247Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6381692Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6383041Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6384363Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6385791Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6387192Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6388954Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6390565Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6392105Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6393653Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6395236Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6397660Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6400157Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6401171Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6402986Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6404529Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6405609Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6406842Z [rank1]:E1204 09:21:13.069000 35755 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.6407829Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6408872Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6410348Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6411811Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6413247Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6414590Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6415944Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6417342Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6418737Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6420148Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6421555Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6422919Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6424729Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6426321Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6428818Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 498008064 and is now 630128640. 2025-12-04T09:22:55.6431107Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6432263Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6434321Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6436079Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6437291Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6438873Z [rank3]:E1204 09:21:13.081000 35757 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.6439738Z dist init r=2, world=4 2025-12-04T09:22:55.6439986Z dist init r=1, world=4 2025-12-04T09:22:55.6440223Z dist init r=3, world=4 2025-12-04T09:22:55.6440785Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6441779Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6443254Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6444695Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6446184Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6447526Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6448861Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6450305Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6451700Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6453306Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6454782Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6456229Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6457677Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6459165Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6461771Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.6464217Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6465456Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6467950Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6469923Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6471268Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6472792Z [rank0]:E1204 09:21:13.227000 35754 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.6473640Z dist init r=0, world=4 2025-12-04T09:22:55.6475093Z [rank0]:[W1204 09:21:13.541643729 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.6476600Z FAILED [8.7646s] [ 33%] 2025-12-04T09:22:55.6476819Z 2025-12-04T09:22:55.6476969Z =================================== FAILURES =================================== 2025-12-04T09:22:55.6477745Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda _ 2025-12-04T09:22:55.6478485Z Traceback (most recent call last): 2025-12-04T09:22:55.6479425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.6480389Z self._join_processes(fn) 2025-12-04T09:22:55.6481222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.6482092Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.6482974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.6483841Z raise RuntimeError(error) 2025-12-04T09:22:55.6484314Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.6484771Z Traceback (most recent call last): 2025-12-04T09:22:55.6485553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6486348Z getattr(self, test_name)() 2025-12-04T09:22:55.6487092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6487874Z fn() 2025-12-04T09:22:55.6488529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6489288Z method(*args, **kwargs) 2025-12-04T09:22:55.6490123Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6490845Z method(*args, **kwargs) 2025-12-04T09:22:55.6491533Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6492243Z with policy(): 2025-12-04T09:22:55.6492893Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6493623Z raise RuntimeError(msg) 2025-12-04T09:22:55.6495108Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6496521Z 2025-12-04T09:22:55.6496710Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6497801Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6498764Z 2025-12-04T09:22:55.6499058Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6499466Z 2025-12-04T09:22:55.6499471Z 2025-12-04T09:22:55.6499673Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.6500213Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.6501328Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-21aecc6f8d181c25.xml - 2025-12-04T09:22:55.6502371Z =========================== short test summary info ============================ 2025-12-04T09:22:55.6503503Z FAILED [8.7646s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.6504666Z Traceback (most recent call last): 2025-12-04T09:22:55.6505361Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6506057Z getattr(self, test_name)() 2025-12-04T09:22:55.6506712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6507429Z fn() 2025-12-04T09:22:55.6508279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6509056Z method(*args, **kwargs) 2025-12-04T09:22:55.6509753Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6510482Z method(*args, **kwargs) 2025-12-04T09:22:55.6511174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6511906Z with policy(): 2025-12-04T09:22:55.6512574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6513327Z raise RuntimeError(msg) 2025-12-04T09:22:55.6514906Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6516364Z 2025-12-04T09:22:55.6516579Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6517727Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6518669Z 2025-12-04T09:22:55.6518931Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6519506Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.6520096Z ======================= 1 failed, 7 deselected in 8.79s ======================== 2025-12-04T09:22:55.6520451Z Got exit code 1 2025-12-04T09:22:55.6520676Z Retrying single test... 2025-12-04T09:22:55.6521383Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-ab70b17c1ec5b8c5.xml 2025-12-04T09:22:55.6522195Z ============================= test session starts ============================== 2025-12-04T09:22:55.6522757Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.6523273Z cachedir: .pytest_cache 2025-12-04T09:22:55.6524196Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.6524952Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.6525400Z configfile: pytest.ini 2025-12-04T09:22:55.6526119Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.6526994Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.6528293Z stepcurrent: skipping 7 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6529425Z Running 1 items in this shard 2025-12-04T09:22:55.6529630Z 2025-12-04T09:22:55.6530801Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda I1204 09:21:19.944000 36039 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 36091 2025-12-04T09:22:55.6532582Z I1204 09:21:19.945000 36039 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 36092 2025-12-04T09:22:55.6533736Z I1204 09:21:19.946000 36039 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 36093 2025-12-04T09:22:55.6534845Z I1204 09:21:19.947000 36039 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 36094 2025-12-04T09:22:55.6537217Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6539044Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6540822Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6542593Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6544347Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6546103Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6548130Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6550136Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6550884Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6552001Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6553662Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6555445Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6557077Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6558602Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6560319Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6561937Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6563418Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6564923Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6566390Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6567857Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6569296Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6570797Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6573052Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6575263Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6576272Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6578087Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6579643Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6580712Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6581938Z [rank1]:E1204 09:21:26.487000 36092 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.6582934Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6583915Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6585439Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6586886Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6595289Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6596880Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6598373Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6600160Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6601562Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6602953Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6604388Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6605744Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6607296Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6608714Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6610847Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6612873Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6613891Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6615718Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6617280Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6618354Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6619583Z [rank2]:E1204 09:21:26.489000 36093 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.6620576Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6621629Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6623100Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6624985Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6626606Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6628217Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6629786Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6631365Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6632937Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6634557Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6636141Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6637668Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6639301Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6640833Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6642957Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 487522304 and is now 630128640. 2025-12-04T09:22:55.6644972Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6645997Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6647812Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6649358Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6650430Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6651730Z [rank3]:E1204 09:21:26.506000 36094 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.6652423Z dist init r=2, world=4 2025-12-04T09:22:55.6652658Z dist init r=1, world=4 2025-12-04T09:22:55.6652895Z dist init r=3, world=4 2025-12-04T09:22:55.6653449Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6654438Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6655909Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6657365Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6658856Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6660199Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6661515Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6662949Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6664358Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6665970Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6667519Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6669208Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6670764Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6672349Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6674767Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.6677048Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6678189Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6680381Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6681939Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6683005Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6684235Z [rank0]:E1204 09:21:26.675000 36091 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.6684919Z dist init r=0, world=4 2025-12-04T09:22:55.6686108Z [rank0]:[W1204 09:21:27.982704130 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.6687322Z FAILED [9.1042s] [100%] 2025-12-04T09:22:55.6687511Z 2025-12-04T09:22:55.6687655Z =================================== FAILURES =================================== 2025-12-04T09:22:55.6688286Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda _ 2025-12-04T09:22:55.6688895Z Traceback (most recent call last): 2025-12-04T09:22:55.6689581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.6690307Z self._join_processes(fn) 2025-12-04T09:22:55.6690995Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.6691752Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.6692521Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.6693262Z raise RuntimeError(error) 2025-12-04T09:22:55.6693652Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.6694074Z Traceback (most recent call last): 2025-12-04T09:22:55.6694749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6695432Z getattr(self, test_name)() 2025-12-04T09:22:55.6696084Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6696749Z fn() 2025-12-04T09:22:55.6697301Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6697956Z method(*args, **kwargs) 2025-12-04T09:22:55.6698571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6699229Z method(*args, **kwargs) 2025-12-04T09:22:55.6699844Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6700495Z with policy(): 2025-12-04T09:22:55.6701090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6701756Z raise RuntimeError(msg) 2025-12-04T09:22:55.6703117Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6704432Z 2025-12-04T09:22:55.6704622Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6705639Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6706480Z 2025-12-04T09:22:55.6706773Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6707123Z 2025-12-04T09:22:55.6707321Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.6707866Z Traceback (most recent call last): 2025-12-04T09:22:55.6708637Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6709429Z getattr(self, test_name)() 2025-12-04T09:22:55.6710160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6710912Z fn() 2025-12-04T09:22:55.6711539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6712273Z method(*args, **kwargs) 2025-12-04T09:22:55.6712973Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6713741Z method(*args, **kwargs) 2025-12-04T09:22:55.6714430Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6715157Z with policy(): 2025-12-04T09:22:55.6715822Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6716605Z raise RuntimeError(msg) 2025-12-04T09:22:55.6718132Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 487522304 and is now 630128640. 2025-12-04T09:22:55.6719600Z 2025-12-04T09:22:55.6719809Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6720963Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6721801Z 2025-12-04T09:22:55.6722035Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6722384Z 2025-12-04T09:22:55.6722388Z 2025-12-04T09:22:55.6722596Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.6723135Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.6724620Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-ab70b17c1ec5b8c5.xml - 2025-12-04T09:22:55.6725723Z =========================== short test summary info ============================ 2025-12-04T09:22:55.6727002Z FAILED [9.1042s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.6728195Z Traceback (most recent call last): 2025-12-04T09:22:55.6728969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6729749Z getattr(self, test_name)() 2025-12-04T09:22:55.6730490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6731239Z fn() 2025-12-04T09:22:55.6731874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6732614Z method(*args, **kwargs) 2025-12-04T09:22:55.6733306Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6734045Z method(*args, **kwargs) 2025-12-04T09:22:55.6734842Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6735580Z with policy(): 2025-12-04T09:22:55.6736235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6737054Z raise RuntimeError(msg) 2025-12-04T09:22:55.6738492Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6739866Z 2025-12-04T09:22:55.6740074Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6741327Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6742494Z 2025-12-04T09:22:55.6742750Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6743194Z 2025-12-04T09:22:55.6743360Z Process 3 exited with error code 10 and exception: 2025-12-04T09:22:55.6743752Z Traceback (most recent call last): 2025-12-04T09:22:55.6744562Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6745496Z getattr(self, test_name)() 2025-12-04T09:22:55.6746284Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6747084Z fn() 2025-12-04T09:22:55.6747991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6748800Z method(*args, **kwargs) 2025-12-04T09:22:55.6749576Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6750379Z method(*args, **kwargs) 2025-12-04T09:22:55.6751150Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6751956Z with policy(): 2025-12-04T09:22:55.6752693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6753508Z raise RuntimeError(msg) 2025-12-04T09:22:55.6755170Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 487522304 and is now 630128640. 2025-12-04T09:22:55.6756779Z 2025-12-04T09:22:55.6756996Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6758288Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6759505Z 2025-12-04T09:22:55.6759760Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6760544Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.6761066Z ======================= 1 failed, 9 deselected in 9.13s ======================== 2025-12-04T09:22:55.6761452Z Got exit code 1 2025-12-04T09:22:55.6761742Z Retrying single test... 2025-12-04T09:22:55.6762555Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e033904d3fffd97a.xml 2025-12-04T09:22:55.6763481Z ============================= test session starts ============================== 2025-12-04T09:22:55.6764231Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.6764839Z cachedir: .pytest_cache 2025-12-04T09:22:55.6765544Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.6766324Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.6766632Z configfile: pytest.ini 2025-12-04T09:22:55.6767371Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.6768255Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.6769541Z stepcurrent: skipping 7 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6770653Z Running 1 items in this shard 2025-12-04T09:22:55.6770955Z 2025-12-04T09:22:55.6772213Z distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda I1204 09:21:33.334000 36376 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 36428 2025-12-04T09:22:55.6773943Z I1204 09:21:33.335000 36376 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 36429 2025-12-04T09:22:55.6775061Z I1204 09:21:33.336000 36376 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 2 with pid 36430 2025-12-04T09:22:55.6776107Z I1204 09:21:33.336000 36376 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 3 with pid 36431 2025-12-04T09:22:55.6778381Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6780340Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6782182Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 2, which does not have an explicit index. FSDP will use the current device 2. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6784033Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6785873Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 3, which does not have an explicit index. FSDP will use the current device 3. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6787868Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6789997Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6792003Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6792749Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6793865Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6795666Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6797305Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6798933Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6800628Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6802029Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6803545Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6805030Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6806531Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6808012Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6809518Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6811089Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6812491Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6814626Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 1. CUDA driver allocated memory was 607059968 and is now 630128640. 2025-12-04T09:22:55.6816644Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6817669Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6819485Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6821036Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6822096Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6823326Z [rank1]:E1204 09:21:39.916000 36429 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.6825028Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6826144Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6827881Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6829518Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6831139Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6832704Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6834190Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6835835Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6837453Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6839025Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6840041Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6840467Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6841364Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6841832Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6843526Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6843874Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6844486Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6845685Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6846026Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6846751Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6847271Z [rank2]:E1204 09:21:39.930000 36430 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 2 with exit code: 10 2025-12-04T09:22:55.6847806Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6848276Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6849157Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6849604Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6850509Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6850857Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6851705Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6852163Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6853013Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6853442Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6854288Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6854682Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6855527Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6855964Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6857787Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 3. CUDA driver allocated memory was 581894144 and is now 630128640. 2025-12-04T09:22:55.6858135Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6858752Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6859944Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6860352Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6861021Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6861534Z [rank3]:E1204 09:21:39.939000 36431 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 3 with exit code: 10 2025-12-04T09:22:55.6861630Z dist init r=1, world=4 2025-12-04T09:22:55.6861723Z dist init r=2, world=4 2025-12-04T09:22:55.6862148Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6862645Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6863596Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6864093Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6865025Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6865428Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6866327Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6866796Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6867933Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6868428Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6869384Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6869832Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6870797Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6871285Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6873066Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 0. CUDA driver allocated memory was 714014720 and is now 739180544. 2025-12-04T09:22:55.6873432Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6874216Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6875484Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6875848Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6876559Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6877112Z [rank0]:E1204 09:21:40.049000 36428 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.6877214Z dist init r=3, world=4 2025-12-04T09:22:55.6877339Z dist init r=0, world=4 2025-12-04T09:22:55.6878503Z [rank0]:[W1204 09:21:40.373105739 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.6878601Z FAILED [8.8841s] [100%] 2025-12-04T09:22:55.6878608Z 2025-12-04T09:22:55.6878757Z =================================== FAILURES =================================== 2025-12-04T09:22:55.6879218Z _ TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda _ 2025-12-04T09:22:55.6879337Z Traceback (most recent call last): 2025-12-04T09:22:55.6879984Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.6880089Z self._join_processes(fn) 2025-12-04T09:22:55.6880649Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.6880796Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.6881382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.6881495Z raise RuntimeError(error) 2025-12-04T09:22:55.6881825Z RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.6881934Z Traceback (most recent call last): 2025-12-04T09:22:55.6882444Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6882546Z getattr(self, test_name)() 2025-12-04T09:22:55.6883156Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6883238Z fn() 2025-12-04T09:22:55.6883684Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6883785Z method(*args, **kwargs) 2025-12-04T09:22:55.6884225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6884312Z method(*args, **kwargs) 2025-12-04T09:22:55.6884756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6884841Z with policy(): 2025-12-04T09:22:55.6885287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6885384Z raise RuntimeError(msg) 2025-12-04T09:22:55.6886643Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6886656Z 2025-12-04T09:22:55.6886883Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6887667Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6887673Z 2025-12-04T09:22:55.6887912Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6887919Z 2025-12-04T09:22:55.6887923Z 2025-12-04T09:22:55.6888118Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.6888347Z Process 2 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.6889056Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e033904d3fffd97a.xml - 2025-12-04T09:22:55.6889239Z =========================== short test summary info ============================ 2025-12-04T09:22:55.6890098Z FAILED [8.8841s] distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda - RuntimeError: Process 2 exited with error code 10 and exception: 2025-12-04T09:22:55.6890204Z Traceback (most recent call last): 2025-12-04T09:22:55.6890691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6890824Z getattr(self, test_name)() 2025-12-04T09:22:55.6891296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6891380Z fn() 2025-12-04T09:22:55.6891824Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6891914Z method(*args, **kwargs) 2025-12-04T09:22:55.6892368Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6892456Z method(*args, **kwargs) 2025-12-04T09:22:55.6892898Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6892984Z with policy(): 2025-12-04T09:22:55.6893428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6893530Z raise RuntimeError(msg) 2025-12-04T09:22:55.6894712Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda! Caching allocator allocated memory was 512 and is now reported as 4608 on device 2. CUDA driver allocated memory was 604962816 and is now 630128640. 2025-12-04T09:22:55.6894719Z 2025-12-04T09:22:55.6894914Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6895638Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestCommunicationCUDA.test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6895642Z 2025-12-04T09:22:55.6895872Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6896033Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.6896192Z ======================= 1 failed, 9 deselected in 8.91s ======================== 2025-12-04T09:22:55.6896277Z Got exit code 1 2025-12-04T09:22:55.6896927Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda 2025-12-04T09:22:55.6897283Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.6897882Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f030d3020f190621.xml 2025-12-04T09:22:55.6898024Z ============================= test session starts ============================== 2025-12-04T09:22:55.6898331Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.6898427Z cachedir: .pytest_cache 2025-12-04T09:22:55.6898879Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.6898991Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.6899082Z configfile: pytest.ini 2025-12-04T09:22:55.6899555Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.6899741Z collecting ... collected 10 items / 8 deselected / 2 selected 2025-12-04T09:22:55.6899886Z stepcurrent: skipping 8 already run items. 2025-12-04T09:22:55.6899988Z Running 2 items in this shard 2025-12-04T09:22:55.6899998Z 2025-12-04T09:22:55.6900911Z distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda I1204 09:21:46.844000 36713 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 36765 2025-12-04T09:22:55.6901347Z I1204 09:21:46.845000 36713 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 36766 2025-12-04T09:22:55.6902914Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6903061Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6904582Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6904724Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6908387Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.6908780Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.6912614Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.6913001Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.6913462Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6913995Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6914997Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6915533Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6916530Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6916924Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6917918Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6918407Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6919368Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6919953Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6920925Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6921323Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6922179Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6922616Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6924438Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.6924804Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6925464Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6926676Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.6927045Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6927752Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6928294Z [rank1]:E1204 09:21:52.218000 36766 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.6928749Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6929276Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6930322Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6930830Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6931816Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6932252Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6933204Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6933702Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6934656Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6935146Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6936099Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6936638Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6937495Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6937925Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6939383Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 0. CUDA driver allocated memory was 453967872 and is now 477036544. 2025-12-04T09:22:55.6939702Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6940338Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6941329Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.6941651Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6942285Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6942763Z [rank0]:E1204 09:21:52.219000 36765 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.6942858Z dist init r=1, world=2 2025-12-04T09:22:55.6942940Z dist init r=0, world=2 2025-12-04T09:22:55.6943994Z [rank0]:[W1204 09:21:52.526709814 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.6944079Z FAILED [7.0448s] [ 50%] 2025-12-04T09:22:55.6944085Z 2025-12-04T09:22:55.6944211Z =================================== FAILURES =================================== 2025-12-04T09:22:55.6944493Z ____ TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda _____ 2025-12-04T09:22:55.6944621Z Traceback (most recent call last): 2025-12-04T09:22:55.6945105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.6945206Z self._join_processes(fn) 2025-12-04T09:22:55.6945721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.6945855Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.6946387Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.6946483Z raise RuntimeError(error) 2025-12-04T09:22:55.6946695Z RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:22:55.6946797Z Traceback (most recent call last): 2025-12-04T09:22:55.6947350Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6947449Z getattr(self, test_name)() 2025-12-04T09:22:55.6948125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6948217Z fn() 2025-12-04T09:22:55.6948717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6948819Z method(*args, **kwargs) 2025-12-04T09:22:55.6949333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6949433Z method(*args, **kwargs) 2025-12-04T09:22:55.6949938Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6950031Z with policy(): 2025-12-04T09:22:55.6950537Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6950646Z raise RuntimeError(msg) 2025-12-04T09:22:55.6951839Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 0. CUDA driver allocated memory was 453967872 and is now 477036544. 2025-12-04T09:22:55.6951847Z 2025-12-04T09:22:55.6952125Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6952791Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.6952797Z 2025-12-04T09:22:55.6953055Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6953060Z 2025-12-04T09:22:55.6953065Z 2025-12-04T09:22:55.6953288Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.6953545Z Process 0 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.6954344Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f030d3020f190621.xml - 2025-12-04T09:22:55.6954510Z =========================== short test summary info ============================ 2025-12-04T09:22:55.6955384Z FAILED [7.0448s] distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda - RuntimeError: Process 0 exited with error code 10 and exception: 2025-12-04T09:22:55.6955513Z Traceback (most recent call last): 2025-12-04T09:22:55.6956061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6956173Z getattr(self, test_name)() 2025-12-04T09:22:55.6956737Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6956820Z fn() 2025-12-04T09:22:55.6957327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6957427Z method(*args, **kwargs) 2025-12-04T09:22:55.6957926Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6958038Z method(*args, **kwargs) 2025-12-04T09:22:55.6958536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6958637Z with policy(): 2025-12-04T09:22:55.6959141Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6959246Z raise RuntimeError(msg) 2025-12-04T09:22:55.6960509Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 0. CUDA driver allocated memory was 453967872 and is now 477036544. 2025-12-04T09:22:55.6960514Z 2025-12-04T09:22:55.6960701Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6961303Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.6961310Z 2025-12-04T09:22:55.6961539Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6961697Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.6961856Z ======================= 1 failed, 8 deselected in 7.07s ======================== 2025-12-04T09:22:55.6961942Z Got exit code 1 2025-12-04T09:22:55.6962043Z Retrying single test... 2025-12-04T09:22:55.6962593Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-da8aed1c14f5aeeb.xml 2025-12-04T09:22:55.6962733Z ============================= test session starts ============================== 2025-12-04T09:22:55.6963048Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.6963140Z cachedir: .pytest_cache 2025-12-04T09:22:55.6963647Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.6963763Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.6963853Z configfile: pytest.ini 2025-12-04T09:22:55.6964333Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.6964514Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.6965178Z stepcurrent: skipping 8 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.6965279Z Running 1 items in this shard 2025-12-04T09:22:55.6965283Z 2025-12-04T09:22:55.6966190Z distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda I1204 09:21:58.134000 36908 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 36960 2025-12-04T09:22:55.6966666Z I1204 09:21:58.135000 36908 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 36961 2025-12-04T09:22:55.6968190Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6968370Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6969887Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.6970035Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.6973410Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.6973771Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.6977126Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.6977526Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.6977934Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6978413Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6979294Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6979740Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6980623Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6980999Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6981850Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6982304Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6983155Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6983583Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6984630Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.6985053Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.6985955Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.6986423Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.6988239Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.6988609Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6989262Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.6990383Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.6990745Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.6991518Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.6992069Z [rank1]:E1204 09:22:03.536000 36961 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.6992515Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.6993043Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.6994038Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.6994539Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.6995568Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.6995961Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.6996921Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6997433Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6998401Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.6998886Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.6999939Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7000359Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7001266Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7001725Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7003454Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 0. CUDA driver allocated memory was 453967872 and is now 477036544. 2025-12-04T09:22:55.7003889Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7004558Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7005803Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7006257Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7007001Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7007573Z [rank0]:E1204 09:22:03.537000 36960 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.7007673Z dist init r=1, world=2 2025-12-04T09:22:55.7007805Z dist init r=0, world=2 2025-12-04T09:22:55.7009020Z [rank0]:[W1204 09:22:03.847318656 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.7009115Z FAILED [6.9428s] [100%] 2025-12-04T09:22:55.7009156Z 2025-12-04T09:22:55.7009358Z =================================== FAILURES =================================== 2025-12-04T09:22:55.7009666Z ____ TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda _____ 2025-12-04T09:22:55.7009780Z Traceback (most recent call last): 2025-12-04T09:22:55.7010383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.7010490Z self._join_processes(fn) 2025-12-04T09:22:55.7011163Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.7011301Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.7011951Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.7012065Z raise RuntimeError(error) 2025-12-04T09:22:55.7012288Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7012470Z Traceback (most recent call last): 2025-12-04T09:22:55.7013005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7013141Z getattr(self, test_name)() 2025-12-04T09:22:55.7013691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7013777Z fn() 2025-12-04T09:22:55.7014339Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7014446Z method(*args, **kwargs) 2025-12-04T09:22:55.7014997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7015092Z method(*args, **kwargs) 2025-12-04T09:22:55.7015652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7015749Z with policy(): 2025-12-04T09:22:55.7016308Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7016411Z raise RuntimeError(msg) 2025-12-04T09:22:55.7017646Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.7017670Z 2025-12-04T09:22:55.7017930Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7018640Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7018647Z 2025-12-04T09:22:55.7019012Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7019078Z 2025-12-04T09:22:55.7019084Z 2025-12-04T09:22:55.7019346Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.7019595Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.7020508Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-da8aed1c14f5aeeb.xml - 2025-12-04T09:22:55.7020880Z =========================== short test summary info ============================ 2025-12-04T09:22:55.7021732Z FAILED [6.9428s] distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7021843Z Traceback (most recent call last): 2025-12-04T09:22:55.7022420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7022564Z getattr(self, test_name)() 2025-12-04T09:22:55.7023095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7023205Z fn() 2025-12-04T09:22:55.7023865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7024127Z method(*args, **kwargs) 2025-12-04T09:22:55.7024763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7024865Z method(*args, **kwargs) 2025-12-04T09:22:55.7025438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7025532Z with policy(): 2025-12-04T09:22:55.7026103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7026219Z raise RuntimeError(msg) 2025-12-04T09:22:55.7027613Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.7027620Z 2025-12-04T09:22:55.7027839Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7028577Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7028583Z 2025-12-04T09:22:55.7028845Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7029059Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.7029271Z ======================= 1 failed, 9 deselected in 6.97s ======================== 2025-12-04T09:22:55.7029368Z Got exit code 1 2025-12-04T09:22:55.7029468Z Retrying single test... 2025-12-04T09:22:55.7030154Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c91f300ede57a5d9.xml 2025-12-04T09:22:55.7030320Z ============================= test session starts ============================== 2025-12-04T09:22:55.7030729Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.7030835Z cachedir: .pytest_cache 2025-12-04T09:22:55.7031401Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.7031529Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.7031637Z configfile: pytest.ini 2025-12-04T09:22:55.7032233Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.7032539Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.7033366Z stepcurrent: skipping 8 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7033477Z Running 1 items in this shard 2025-12-04T09:22:55.7033482Z 2025-12-04T09:22:55.7034651Z distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda I1204 09:22:09.504000 37103 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 37155 2025-12-04T09:22:55.7035175Z I1204 09:22:09.505000 37103 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 37156 2025-12-04T09:22:55.7037051Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.7037274Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.7039312Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.7039506Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.7043188Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.7043545Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.7047021Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.7047368Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.7047773Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.7048247Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.7049191Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7049648Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.7050592Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7050947Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.7051801Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7052262Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7053115Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7053569Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7054424Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7054816Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7055676Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7056106Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7057648Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 0. CUDA driver allocated memory was 453967872 and is now 477036544. 2025-12-04T09:22:55.7057978Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7058563Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7059568Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7059889Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7060527Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7061004Z [rank0]:E1204 09:22:14.906000 37155 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.7061469Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.7061942Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.7062822Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7063277Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.7064145Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7064501Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.7065382Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7065814Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7066690Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7067119Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7068256Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7068704Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7069674Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7070163Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7071896Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.7072276Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7072929Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7074059Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7074421Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7075142Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7075749Z [rank1]:E1204 09:22:14.906000 37156 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.7075853Z dist init r=1, world=2 2025-12-04T09:22:55.7075954Z dist init r=0, world=2 2025-12-04T09:22:55.7077108Z [rank0]:[W1204 09:22:15.223950719 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.7077211Z FAILED [6.8171s] [100%] 2025-12-04T09:22:55.7077217Z 2025-12-04T09:22:55.7077362Z =================================== FAILURES =================================== 2025-12-04T09:22:55.7077671Z ____ TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda _____ 2025-12-04T09:22:55.7077795Z Traceback (most recent call last): 2025-12-04T09:22:55.7078343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.7078490Z self._join_processes(fn) 2025-12-04T09:22:55.7079070Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.7079210Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.7079919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.7080236Z raise RuntimeError(error) 2025-12-04T09:22:55.7080442Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7080549Z Traceback (most recent call last): 2025-12-04T09:22:55.7081027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7081131Z getattr(self, test_name)() 2025-12-04T09:22:55.7081610Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7081685Z fn() 2025-12-04T09:22:55.7082142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7082235Z method(*args, **kwargs) 2025-12-04T09:22:55.7082677Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7082772Z method(*args, **kwargs) 2025-12-04T09:22:55.7083213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7083304Z with policy(): 2025-12-04T09:22:55.7083748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7083841Z raise RuntimeError(msg) 2025-12-04T09:22:55.7084915Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.7084921Z 2025-12-04T09:22:55.7085109Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7085714Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7085720Z 2025-12-04T09:22:55.7085955Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7085960Z 2025-12-04T09:22:55.7085964Z 2025-12-04T09:22:55.7086163Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.7086396Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.7087152Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c91f300ede57a5d9.xml - 2025-12-04T09:22:55.7087313Z =========================== short test summary info ============================ 2025-12-04T09:22:55.7088047Z FAILED [6.8171s] distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7088156Z Traceback (most recent call last): 2025-12-04T09:22:55.7088639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7088734Z getattr(self, test_name)() 2025-12-04T09:22:55.7089210Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7089286Z fn() 2025-12-04T09:22:55.7089733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7089854Z method(*args, **kwargs) 2025-12-04T09:22:55.7090295Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7090386Z method(*args, **kwargs) 2025-12-04T09:22:55.7090829Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7090937Z with policy(): 2025-12-04T09:22:55.7091390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7091482Z raise RuntimeError(msg) 2025-12-04T09:22:55.7092552Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda! Caching allocator allocated memory was 512 and is now reported as 13824 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.7092565Z 2025-12-04T09:22:55.7092752Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7093341Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7093346Z 2025-12-04T09:22:55.7093583Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7093741Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.7093901Z ======================= 1 failed, 9 deselected in 6.84s ======================== 2025-12-04T09:22:55.7093981Z Got exit code 1 2025-12-04T09:22:55.7094497Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda 2025-12-04T09:22:55.7094862Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.7095416Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a2a9be50a03cddfc.xml 2025-12-04T09:22:55.7095557Z ============================= test session starts ============================== 2025-12-04T09:22:55.7095874Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.7095965Z cachedir: .pytest_cache 2025-12-04T09:22:55.7096421Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.7096528Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.7096618Z configfile: pytest.ini 2025-12-04T09:22:55.7097161Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.7097342Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.7097522Z stepcurrent: skipping 9 already run items. 2025-12-04T09:22:55.7097623Z Running 1 items in this shard 2025-12-04T09:22:55.7097628Z 2025-12-04T09:22:55.7098546Z distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda I1204 09:22:20.864000 37298 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 37350 2025-12-04T09:22:55.7098986Z I1204 09:22:20.865000 37298 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 37351 2025-12-04T09:22:55.7100504Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.7100687Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.7102203Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.7102392Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.7105753Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.7106105Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.7109940Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.7110339Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.7110801Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.7111340Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.7112399Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7112908Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.7113899Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7114292Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.7115255Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7115740Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7116736Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7117216Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7118165Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7118645Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7119713Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7120185Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7121727Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 0. CUDA driver allocated memory was 453967872 and is now 477036544. 2025-12-04T09:22:55.7122080Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7122687Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7123887Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7124415Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7125124Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7125678Z [rank0]:E1204 09:22:26.267000 37350 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.7126127Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.7126661Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.7127758Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7128260Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.7129254Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7129651Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.7130622Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7131138Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7132100Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7132615Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7133578Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7134026Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7134992Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7135484Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7137245Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.7137575Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7138160Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7139142Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7139464Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7140096Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7140576Z [rank1]:E1204 09:22:26.269000 37351 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.7140664Z dist init r=1, world=2 2025-12-04T09:22:55.7140750Z dist init r=0, world=2 2025-12-04T09:22:55.7141821Z [rank0]:[W1204 09:22:26.581695494 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.7141907Z FAILED [7.1341s] [100%] 2025-12-04T09:22:55.7141911Z 2025-12-04T09:22:55.7142046Z =================================== FAILURES =================================== 2025-12-04T09:22:55.7142319Z _____ TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda _____ 2025-12-04T09:22:55.7142423Z Traceback (most recent call last): 2025-12-04T09:22:55.7142909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.7143005Z self._join_processes(fn) 2025-12-04T09:22:55.7143527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.7143682Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.7144213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.7144315Z raise RuntimeError(error) 2025-12-04T09:22:55.7144519Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7144623Z Traceback (most recent call last): 2025-12-04T09:22:55.7145124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7145217Z getattr(self, test_name)() 2025-12-04T09:22:55.7145691Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7145769Z fn() 2025-12-04T09:22:55.7146213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7146316Z method(*args, **kwargs) 2025-12-04T09:22:55.7146764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7146863Z method(*args, **kwargs) 2025-12-04T09:22:55.7147362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7147449Z with policy(): 2025-12-04T09:22:55.7148106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7148210Z raise RuntimeError(msg) 2025-12-04T09:22:55.7149549Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.7149565Z 2025-12-04T09:22:55.7149780Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7150447Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7150453Z 2025-12-04T09:22:55.7150718Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7150724Z 2025-12-04T09:22:55.7150730Z 2025-12-04T09:22:55.7150944Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.7151203Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.7152001Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a2a9be50a03cddfc.xml - 2025-12-04T09:22:55.7152164Z =========================== short test summary info ============================ 2025-12-04T09:22:55.7153068Z FAILED [7.1341s] distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7153187Z Traceback (most recent call last): 2025-12-04T09:22:55.7153740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7153849Z getattr(self, test_name)() 2025-12-04T09:22:55.7154380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7154471Z fn() 2025-12-04T09:22:55.7154974Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7155073Z method(*args, **kwargs) 2025-12-04T09:22:55.7155580Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7155725Z method(*args, **kwargs) 2025-12-04T09:22:55.7156229Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7156323Z with policy(): 2025-12-04T09:22:55.7156827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7156968Z raise RuntimeError(msg) 2025-12-04T09:22:55.7158167Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 342818816 and is now 367984640. 2025-12-04T09:22:55.7158173Z 2025-12-04T09:22:55.7158391Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7159060Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7159068Z 2025-12-04T09:22:55.7159326Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7159617Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.7159783Z ======================= 1 failed, 9 deselected in 7.16s ======================== 2025-12-04T09:22:55.7159881Z Got exit code 1 2025-12-04T09:22:55.7160098Z Retrying single test... 2025-12-04T09:22:55.7160641Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-be7435212e20eea8.xml 2025-12-04T09:22:55.7160786Z ============================= test session starts ============================== 2025-12-04T09:22:55.7161094Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.7161186Z cachedir: .pytest_cache 2025-12-04T09:22:55.7161649Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.7161753Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.7161850Z configfile: pytest.ini 2025-12-04T09:22:55.7162320Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.7162502Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.7163172Z stepcurrent: skipping 9 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7163269Z Running 1 items in this shard 2025-12-04T09:22:55.7163273Z 2025-12-04T09:22:55.7164238Z distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda I1204 09:22:32.154000 37493 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 37545 2025-12-04T09:22:55.7164680Z I1204 09:22:32.155000 37493 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 37546 2025-12-04T09:22:55.7166212Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.7166362Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.7168103Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.7168292Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.7171836Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.7172250Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.7175808Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.7176177Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.7176610Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.7177109Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.7178050Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7178531Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.7179515Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7179893Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.7180787Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7181250Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7182144Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7182594Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7183539Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7183955Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7184864Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7185349Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7186894Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 0. CUDA driver allocated memory was 453967872 and is now 477036544. 2025-12-04T09:22:55.7187302Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7188100Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7189216Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7189575Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7190299Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7190843Z [rank0]:E1204 09:22:37.483000 37545 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.7191296Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.7191824Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.7192824Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7193335Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.7194381Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7194779Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.7195737Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7196227Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7197188Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7197697Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7198659Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7199129Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7200285Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7200717Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7202183Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 347013120 and is now 367984640. 2025-12-04T09:22:55.7202506Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7203084Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7204070Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7204390Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7205029Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7205505Z [rank1]:E1204 09:22:37.485000 37546 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.7205593Z dist init r=1, world=2 2025-12-04T09:22:55.7205683Z dist init r=0, world=2 2025-12-04T09:22:55.7206699Z [rank0]:[W1204 09:22:37.803629682 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.7206789Z FAILED [6.9848s] [100%] 2025-12-04T09:22:55.7206795Z 2025-12-04T09:22:55.7206985Z =================================== FAILURES =================================== 2025-12-04T09:22:55.7207265Z _____ TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda _____ 2025-12-04T09:22:55.7207372Z Traceback (most recent call last): 2025-12-04T09:22:55.7207858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.7207963Z self._join_processes(fn) 2025-12-04T09:22:55.7208483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.7208606Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.7209143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.7209242Z raise RuntimeError(error) 2025-12-04T09:22:55.7209476Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7209591Z Traceback (most recent call last): 2025-12-04T09:22:55.7210066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7210168Z getattr(self, test_name)() 2025-12-04T09:22:55.7210638Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7210742Z fn() 2025-12-04T09:22:55.7211195Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7211285Z method(*args, **kwargs) 2025-12-04T09:22:55.7211726Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7211821Z method(*args, **kwargs) 2025-12-04T09:22:55.7212267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7212362Z with policy(): 2025-12-04T09:22:55.7212809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7212905Z raise RuntimeError(msg) 2025-12-04T09:22:55.7213965Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 347013120 and is now 367984640. 2025-12-04T09:22:55.7213972Z 2025-12-04T09:22:55.7214157Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7214756Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7214763Z 2025-12-04T09:22:55.7214998Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7215003Z 2025-12-04T09:22:55.7215007Z 2025-12-04T09:22:55.7215202Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.7215428Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.7216135Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-be7435212e20eea8.xml - 2025-12-04T09:22:55.7216294Z =========================== short test summary info ============================ 2025-12-04T09:22:55.7217019Z FAILED [6.9848s] distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7217131Z Traceback (most recent call last): 2025-12-04T09:22:55.7217671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7217772Z getattr(self, test_name)() 2025-12-04T09:22:55.7218252Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7218329Z fn() 2025-12-04T09:22:55.7218772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7218871Z method(*args, **kwargs) 2025-12-04T09:22:55.7219311Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7219405Z method(*args, **kwargs) 2025-12-04T09:22:55.7219846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7219929Z with policy(): 2025-12-04T09:22:55.7220395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7220519Z raise RuntimeError(msg) 2025-12-04T09:22:55.7221576Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 347013120 and is now 367984640. 2025-12-04T09:22:55.7221581Z 2025-12-04T09:22:55.7221797Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7222387Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7222392Z 2025-12-04T09:22:55.7222623Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7222777Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.7222940Z ======================= 1 failed, 9 deselected in 7.01s ======================== 2025-12-04T09:22:55.7223027Z Got exit code 1 2025-12-04T09:22:55.7223116Z Retrying single test... 2025-12-04T09:22:55.7223977Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc23628278ddca.xml 2025-12-04T09:22:55.7224127Z ============================= test session starts ============================== 2025-12-04T09:22:55.7224635Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.7224742Z cachedir: .pytest_cache 2025-12-04T09:22:55.7225255Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.7225380Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.7225482Z configfile: pytest.ini 2025-12-04T09:22:55.7226020Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.7226232Z collecting ... collected 10 items / 9 deselected / 1 selected 2025-12-04T09:22:55.7226979Z stepcurrent: skipping 9 already run items. Running only test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7227091Z Running 1 items in this shard 2025-12-04T09:22:55.7227096Z 2025-12-04T09:22:55.7228220Z distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda I1204 09:22:43.394000 37688 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 0 with pid 37740 2025-12-04T09:22:55.7228716Z I1204 09:22:43.395000 37688 site-packages/torch/testing/_internal/common_distributed.py:849] Started process 1 with pid 37741 2025-12-04T09:22:55.7230538Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.7230706Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.7232417Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2025-12-04T09:22:55.7232581Z device_from_device_id = _get_device_from_device_id( 2025-12-04T09:22:55.7236382Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.7236864Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.7240611Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py:865: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:76.) 2025-12-04T09:22:55.7240958Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-12-04T09:22:55.7241371Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.7241844Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.7242738Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7243188Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.7244075Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7244424Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.7245325Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7245764Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7246611Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7247046Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7247886Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7248286Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7249160Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7249592Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7251073Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 0. CUDA driver allocated memory was 453967872 and is now 477036544. 2025-12-04T09:22:55.7251396Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7251986Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7252972Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7253304Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7253937Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7254420Z [rank0]:E1204 09:22:48.752000 37740 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 0 with exit code: 10 2025-12-04T09:22:55.7254827Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] Caught exception: 2025-12-04T09:22:55.7255296Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] Traceback (most recent call last): 2025-12-04T09:22:55.7256184Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7256628Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] getattr(self, test_name)() 2025-12-04T09:22:55.7257507Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7257922Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] fn() 2025-12-04T09:22:55.7258769Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7259205Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7260056Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7260489Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] method(*args, **kwargs) 2025-12-04T09:22:55.7261344Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7261772Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] with policy(): 2025-12-04T09:22:55.7262622Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7263079Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] raise RuntimeError(msg) 2025-12-04T09:22:55.7264537Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 347013120 and is now 367984640. 2025-12-04T09:22:55.7264859Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7265444Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7266433Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7266760Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] 2025-12-04T09:22:55.7267449Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7268144Z [rank1]:E1204 09:22:48.753000 37741 site-packages/torch/testing/_internal/common_distributed.py:935] exiting process 1 with exit code: 10 2025-12-04T09:22:55.7268250Z dist init r=1, world=2 2025-12-04T09:22:55.7268347Z dist init r=0, world=2 2025-12-04T09:22:55.7269511Z [rank0]:[W1204 09:22:49.066802844 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-12-04T09:22:55.7269612Z FAILED [6.7779s] [100%] 2025-12-04T09:22:55.7269617Z 2025-12-04T09:22:55.7269761Z =================================== FAILURES =================================== 2025-12-04T09:22:55.7270073Z _____ TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda _____ 2025-12-04T09:22:55.7270188Z Traceback (most recent call last): 2025-12-04T09:22:55.7270800Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper 2025-12-04T09:22:55.7270912Z self._join_processes(fn) 2025-12-04T09:22:55.7271492Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes 2025-12-04T09:22:55.7271642Z self._check_return_codes(fn, elapsed_time) 2025-12-04T09:22:55.7272243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes 2025-12-04T09:22:55.7272355Z raise RuntimeError(error) 2025-12-04T09:22:55.7272589Z RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7272709Z Traceback (most recent call last): 2025-12-04T09:22:55.7273254Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7273362Z getattr(self, test_name)() 2025-12-04T09:22:55.7273924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7274017Z fn() 2025-12-04T09:22:55.7274520Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7274618Z method(*args, **kwargs) 2025-12-04T09:22:55.7275125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7275256Z method(*args, **kwargs) 2025-12-04T09:22:55.7275765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7275857Z with policy(): 2025-12-04T09:22:55.7276362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7276476Z raise RuntimeError(msg) 2025-12-04T09:22:55.7277663Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 347013120 and is now 367984640. 2025-12-04T09:22:55.7277670Z 2025-12-04T09:22:55.7277890Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7278553Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7278561Z 2025-12-04T09:22:55.7278822Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7278828Z 2025-12-04T09:22:55.7278840Z 2025-12-04T09:22:55.7279053Z ----------------------------- Captured stdout call ----------------------------- 2025-12-04T09:22:55.7279310Z Process 1 terminated with exit code 10, terminating remaining processes. 2025-12-04T09:22:55.7280210Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc23628278ddca.xml - 2025-12-04T09:22:55.7280359Z =========================== short test summary info ============================ 2025-12-04T09:22:55.7281079Z FAILED [6.7779s] distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda - RuntimeError: Process 1 exited with error code 10 and exception: 2025-12-04T09:22:55.7281193Z Traceback (most recent call last): 2025-12-04T09:22:55.7281679Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test 2025-12-04T09:22:55.7281785Z getattr(self, test_name)() 2025-12-04T09:22:55.7282256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper 2025-12-04T09:22:55.7282332Z fn() 2025-12-04T09:22:55.7283103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7283199Z method(*args, **kwargs) 2025-12-04T09:22:55.7283652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper 2025-12-04T09:22:55.7283741Z method(*args, **kwargs) 2025-12-04T09:22:55.7284184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3328, in wrapper 2025-12-04T09:22:55.7284271Z with policy(): 2025-12-04T09:22:55.7284717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2705, in __exit__ 2025-12-04T09:22:55.7284809Z raise RuntimeError(msg) 2025-12-04T09:22:55.7285873Z RuntimeError: CUDA driver API confirmed a leak in __mp_main__.TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda! Caching allocator allocated memory was 512 and is now reported as 9216 on device 1. CUDA driver allocated memory was 347013120 and is now 367984640. 2025-12-04T09:22:55.7285971Z 2025-12-04T09:22:55.7286160Z To execute this test, run the following from the base repo dir: 2025-12-04T09:22:55.7286758Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=1 python test/distributed/fsdp/test_fsdp_comm.py TestExplicitUnshardCUDA.test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7286763Z 2025-12-04T09:22:55.7286995Z This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0 2025-12-04T09:22:55.7287184Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! 2025-12-04T09:22:55.7287336Z ======================= 1 failed, 9 deselected in 6.80s ======================== 2025-12-04T09:22:55.7287418Z Got exit code 1 2025-12-04T09:22:55.7287941Z FAILED CONSISTENTLY: test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda 2025-12-04T09:22:55.7288301Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set 2025-12-04T09:22:55.7288855Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e7d51fdd1a2eae54.xml 2025-12-04T09:22:55.7289001Z ============================= test session starts ============================== 2025-12-04T09:22:55.7289305Z platform linux -- Python 3.10.14, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-12-04T09:22:55.7289405Z cachedir: .pytest_cache 2025-12-04T09:22:55.7289857Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-12-04T09:22:55.7289962Z rootdir: /var/lib/jenkins/workspace 2025-12-04T09:22:55.7290060Z configfile: pytest.ini 2025-12-04T09:22:55.7290529Z plugins: hypothesis-6.56.4, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.3.0, typeguard-4.3.0 2025-12-04T09:22:55.7290713Z collecting ... collected 10 items / 10 deselected / 0 selected 2025-12-04T09:22:55.7290848Z stepcurrent: skipping 10 already run items. 2025-12-04T09:22:55.7290946Z Running 0 items in this shard 2025-12-04T09:22:55.7290951Z 2025-12-04T09:22:55.7291667Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e7d51fdd1a2eae54.xml - 2025-12-04T09:22:55.7291809Z ============================ 10 deselected in 0.01s ============================ 2025-12-04T09:22:55.7298207Z The following tests failed consistently: ['test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda', 'test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda'] 2025-12-04T09:22:55.7298220Z 2025-12-04T09:22:55.7298827Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm 1/1 (test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_8c711ca16ddf8d8f_.log) 2025-12-04T09:22:55.7298833Z 2025-12-04T09:22:55.7299177Z Finished distributed/fsdp/test_fsdp_comm 1/1 ... [2025-12-04 09:22:55.344870][1806.28250154], took 6.61min 2025-12-04T09:22:55.7300143Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a25290001c1b014e.xml 2025-12-04T09:22:55.7301019Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-cc80cfecc5abaf73.xml 2025-12-04T09:22:55.7301858Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-00aad17389ddcea3.xml 2025-12-04T09:22:55.7302688Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc7285fa7f4227.xml 2025-12-04T09:22:55.7303498Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-69d9e5e2bc4a3a7d.xml 2025-12-04T09:22:55.7304310Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c1c17128b5541633.xml 2025-12-04T09:22:55.7305121Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-156a51594e715e30.xml 2025-12-04T09:22:55.7306055Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2e6a295727102ba7.xml 2025-12-04T09:22:55.7306980Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2bb3299d57ce6c2b.xml 2025-12-04T09:22:55.7308099Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-b434b54a558efa89.xml 2025-12-04T09:22:55.7309511Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-bd75c8cd2cf8ff9e.xml 2025-12-04T09:22:55.7615726Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f7fa0727268f1a56.xml 2025-12-04T09:22:55.7908291Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f90badc543da9251.xml 2025-12-04T09:22:55.8193982Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-31be29224184e218.xml 2025-12-04T09:22:55.8666145Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1c567b53e42c8343.xml 2025-12-04T09:22:55.8985716Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-3db6e54b58890527.xml 2025-12-04T09:22:55.9271587Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-5f8a5af0d826239e.xml 2025-12-04T09:22:55.9547787Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1a3a2c29e874821d.xml 2025-12-04T09:22:55.9950623Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-097bd188165cd189.xml 2025-12-04T09:22:56.0270674Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-efc9fa1f0bd357d6.xml 2025-12-04T09:22:56.0569339Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-648e0de30361d669.xml 2025-12-04T09:22:56.0854663Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-21aecc6f8d181c25.xml 2025-12-04T09:22:56.1147007Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-ab70b17c1ec5b8c5.xml 2025-12-04T09:22:56.1426067Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e033904d3fffd97a.xml 2025-12-04T09:22:56.1712972Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f030d3020f190621.xml 2025-12-04T09:22:56.2026213Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-da8aed1c14f5aeeb.xml 2025-12-04T09:22:56.2323146Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c91f300ede57a5d9.xml 2025-12-04T09:22:56.2607946Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a2a9be50a03cddfc.xml 2025-12-04T09:22:56.2888950Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-be7435212e20eea8.xml 2025-12-04T09:22:56.3188468Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc23628278ddca.xml 2025-12-04T09:22:56.3426684Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e7d51fdd1a2eae54.xml 2025-12-04T09:22:56.5868855Z Uploading logs for 57116084912 to S3 2025-12-04T09:22:56.6354175Z Uploading artifacts took 0.27 seconds 2025-12-04T09:22:56.6354962Z distributed/fsdp/test_fsdp_comm 1/1 failed! 2025-12-04T09:22:56.6357105Z Running distributed/fsdp/test_distributed_checkpoint 1/1 ... [2025-12-04 09:22:56.635587][1807.573224425] 2025-12-04T09:22:56.6357767Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:22:56.6361079Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:22:56.635903] 2025-12-04T09:23:00.8109852Z 2025-12-04T09:23:00.8111123Z distributed/fsdp/test_distributed_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_6287b63322acae31_.log 2025-12-04T09:23:00.8113739Z Running 2 items in this shard: test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type0_cuda, test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type1_cuda 2025-12-04T09:23:00.8115331Z 2025-12-04T09:23:00.8115882Z Finished distributed/fsdp/test_distributed_checkpoint 1/1 ... [2025-12-04 09:23:00.810594][1811.748229829], took 0.07min 2025-12-04T09:23:00.8177988Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-19176bf0e30ddda9.xml 2025-12-04T09:23:00.8500351Z Running distributed/test_c10d_ops_nccl 1/1 ... [2025-12-04 09:23:00.849384][1811.787021837] 2025-12-04T09:23:00.8500955Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:23:00.8502346Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ops_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:23:00.849729] 2025-12-04T09:23:14.0982832Z 2025-12-04T09:23:14.0983853Z distributed/test_c10d_ops_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ops_nccl_1.1_fe8272077a352f98_.log 2025-12-04T09:23:14.0998245Z Running 30 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_all_gather_v, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_float8, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_bfloat16, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_float8, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_v, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2025-12-04T09:23:14.1010913Z 2025-12-04T09:23:14.1011279Z Finished distributed/test_c10d_ops_nccl 1/1 ... [2025-12-04 09:23:14.097702][1825.035337762], took 0.22min 2025-12-04T09:23:14.1049580Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-a72c588f9917f825.xml 2025-12-04T09:23:14.2050717Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2025-12-04 09:23:14.204393][1825.142030233] 2025-12-04T09:23:14.2051432Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:23:14.2052784Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:23:14.204734] 2025-12-04T09:23:46.9038335Z 2025-12-04T09:23:46.9039735Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_e759e0b14b68a94b_.log 2025-12-04T09:23:46.9055334Z Running 26 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait_on_a_child_thread, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_wait_for_all_child_procs_to_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsFuncTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary_duplicate_log_filters, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary_redirect_and_tee 2025-12-04T09:23:46.9069913Z 2025-12-04T09:23:46.9070410Z Finished distributed/elastic/multiprocessing/api_test 1/1 ... [2025-12-04 09:23:46.903310][1857.840939245], took 0.54min 2025-12-04T09:23:46.9108125Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-3223bbc58a6df58f.xml 2025-12-04T09:23:46.9949923Z Running distributed/test_inductor_collectives 1/1 ... [2025-12-04 09:23:46.994599][1857.932235619] 2025-12-04T09:23:46.9950804Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:23:46.9952586Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:23:46.995032] 2025-12-04T09:30:12.7669914Z 2025-12-04T09:30:12.7671069Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_4b073d51b494bfec_.log 2025-12-04T09:30:12.7711949Z Running 69 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_recompute_is_always_banned_override_with_ac_False, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_recompute_is_always_banned_override_with_ac_True, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_scalar_tensor_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_async_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_all_gather_bucket_bucket_mode_all, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_all_gather_bucket_bucket_mode_all_custom_ops, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_all_gather_bucket_multidtype_bucket_mode_all_custom_ops_multidtype, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_all_gather_bucket_path, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_all_reduce_bucket_bucket_mode_all, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared_graph_partition, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reduce_scatter_bucket_bucket_mode_all, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reduce_scatter_bucket_bucket_mode_all_custom_ops, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_peak_memory, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_peak_memory_bucketed_bucket_mode_all, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_peak_memory_bucketed_bucket_mode_all_custom_ops, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_respects_wait_dep, test/distributed/test_inductor_collectives.py::TestSyncDecisionCrossRanks::test_all_gather_comm_analysis, test/distributed/test_inductor_collectives.py::TestSyncDecisionCrossRanks::test_all_reduce_comm_analysis, test/distributed/test_inductor_collectives.py::TestSyncDecisionCrossRanks::test_all_to_all_comm_analysis, test/distributed/test_inductor_collectives.py::TestSyncDecisionCrossRanks::test_reduce_scatter_comm_analysis, test/distributed/test_inductor_collectives.py::TestSyncDecisionCrossRanks::test_regression_use_nccl_estimate_with_gloo, test/distributed/test_inductor_collectives.py::TestSyncDecisionCrossRanks::test_sync_decision_cross_ranks 2025-12-04T09:30:12.7752178Z 2025-12-04T09:30:12.7752610Z Finished distributed/test_inductor_collectives 1/1 ... [2025-12-04 09:30:12.766837][2243.704471698], took 6.43min 2025-12-04T09:30:12.7754186Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-c6dfa06eb71f87cd.xml 2025-12-04T09:30:12.8648654Z Running distributed/tensor/test_dtensor_export 1/1 ... [2025-12-04 09:30:12.864214][2243.801850707] 2025-12-04T09:30:12.8649303Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:30:12.8650613Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_dtensor_export.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:30:12.864556] 2025-12-04T09:30:22.4022557Z 2025-12-04T09:30:22.4024009Z distributed/tensor/test_dtensor_export 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_export_1.1_1c7ff51f7e217adf_.log 2025-12-04T09:30:22.4030641Z Running 9 items in this shard: test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_annotate_aot_export_joint_with_descriptors_alone, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_dtensor_data_dependent_index_and_slice, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_dynamic_shapes_export_fn_with_answer0, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_einsum_dtensor_export_export_fn0, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_export_parallelize_module_with_dtensor_input_export_fn0, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_export_parallelize_module_with_dtensor_input_export_fn1, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_flex_attention_dtensor_export_export_fn0, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_strict_export_parallelize_module_with_dtensor_input, test/distributed/tensor/test_dtensor_export.py::DTensorExportTest::test_union_typed_annotation 2025-12-04T09:30:22.4036510Z 2025-12-04T09:30:22.4036890Z Finished distributed/tensor/test_dtensor_export 1/1 ... [2025-12-04 09:30:22.401594][2253.339229544], took 0.16min 2025-12-04T09:30:22.4089098Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-8b65af0c5c32155a.xml 2025-12-04T09:30:22.4687602Z Running distributed/_tools/test_fake_collectives 1/1 ... [2025-12-04 09:30:22.468211][2253.405848377] 2025-12-04T09:30:22.4688262Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:30:22.4689543Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_fake_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:30:22.468551] 2025-12-04T09:30:26.5939122Z 2025-12-04T09:30:26.5940281Z distributed/_tools/test_fake_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_fake_collectives_1.1_1ec2dba68452f8d1_.log 2025-12-04T09:30:26.5941767Z Running 1 items in this shard: test/distributed/_tools/test_fake_collectives.py::TestFakeCollectives::test_collectives 2025-12-04T09:30:26.5942382Z 2025-12-04T09:30:26.5942802Z Finished distributed/_tools/test_fake_collectives 1/1 ... [2025-12-04 09:30:26.593485][2257.531106612], took 0.07min 2025-12-04T09:30:26.6009070Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._tools.test_fake_collectives/distributed._tools.test_fake_collectives-619983be33386a60.xml 2025-12-04T09:30:26.6318995Z Running distributed/test_control_collectives 1/1 ... [2025-12-04 09:30:26.631299][2257.568936406] 2025-12-04T09:30:26.6319636Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:30:26.6320916Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_control_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:30:26.631675] 2025-12-04T09:30:30.5060362Z 2025-12-04T09:30:30.5061493Z distributed/test_control_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_control_collectives_1.1_1b00fbe408668a80_.log 2025-12-04T09:30:30.5068002Z Running 13 items in this shard: test/distributed/test_control_collectives.py::TestCollectives::test_all_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_barrier, test/distributed/test_control_collectives.py::TestCollectives::test_barrier_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_gather, test/distributed/test_control_collectives.py::TestCollectives::test_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_scatter, test/distributed/test_control_collectives.py::TestCollectives::test_scatter_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_simple_user_func, test/distributed/test_control_collectives.py::TestCollectives::test_unique 2025-12-04T09:30:30.5073962Z 2025-12-04T09:30:30.5074352Z Finished distributed/test_control_collectives 1/1 ... [2025-12-04 09:30:30.505545][2261.443179925], took 0.06min 2025-12-04T09:30:30.5130157Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_control_collectives/distributed.test_control_collectives-3f2bbee5075aa12d.xml 2025-12-04T09:30:30.5478748Z Running distributed/test_collective_utils 1/1 ... [2025-12-04 09:30:30.547295][2261.484931409] 2025-12-04T09:30:30.5479361Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:30:30.5480607Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:30:30.547657] 2025-12-04T09:31:07.1558990Z 2025-12-04T09:31:07.1561141Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_39bf695810dbd3dc_.log 2025-12-04T09:31:07.1567118Z Running 9 items in this shard: test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result_no_pg, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result_raises_exceptions_from_func, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result_no_pg, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result_raises_exceptions_from_func, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_check_rng_sync_device_cpu, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_check_rng_sync_device_cuda, test/distributed/test_collective_utils.py::TestUtils::test_summarize_ranks 2025-12-04T09:31:07.1571525Z 2025-12-04T09:31:07.1571930Z Finished distributed/test_collective_utils 1/1 ... [2025-12-04 09:31:07.155817][2298.093452128], took 0.61min 2025-12-04T09:31:07.1635838Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_collective_utils/distributed.test_collective_utils-9394cde5cb4e06c2.xml 2025-12-04T09:31:07.2360160Z Running distributed/algorithms/quantization/test_quantization 1/1 ... [2025-12-04 09:31:07.235506][2298.173144018] 2025-12-04T09:31:07.2361045Z Running distributed tests for the test backend with env init_method 2025-12-04T09:31:07.2361643Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:31:07.2365831Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:31:07.236359] 2025-12-04T09:31:10.5580754Z 2025-12-04T09:31:10.5582153Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_dc7c5eed900a4a9e_.log 2025-12-04T09:31:10.5583470Z Running 0 items in this shard: 2025-12-04T09:31:10.5583690Z 2025-12-04T09:31:10.5586053Z Running distributed tests for the test backend with file init_method 2025-12-04T09:31:10.5588049Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:31:10.5592545Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:31:10.559028] 2025-12-04T09:31:13.8775765Z 2025-12-04T09:31:13.8777171Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_2349b206b6fb541a_.log 2025-12-04T09:31:13.8778477Z Running 0 items in this shard: 2025-12-04T09:31:13.8778691Z 2025-12-04T09:31:13.8781682Z Running distributed tests for the mpi backend with env init_method 2025-12-04T09:31:14.0109659Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:31:14.0111435Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:31:14.010833] 2025-12-04T09:31:17.9584975Z 2025-12-04T09:31:17.9586793Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_07b4c07c44ff8426_.log 2025-12-04T09:31:17.9588492Z Running 0 items in this shard: 2025-12-04T09:31:17.9588821Z Running 0 items in this shard: 2025-12-04T09:31:17.9589155Z Running 0 items in this shard: 2025-12-04T09:31:17.9589364Z 2025-12-04T09:31:17.9590492Z Running distributed tests for the mpi backend with file init_method 2025-12-04T09:31:18.0809891Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:31:18.0811616Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:31:18.080767] 2025-12-04T09:31:22.0185802Z 2025-12-04T09:31:22.0187227Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_ce2c4a7056f6918d_.log 2025-12-04T09:31:22.0189136Z Running 0 items in this shard: 2025-12-04T09:31:22.0189481Z Running 0 items in this shard: 2025-12-04T09:31:22.0189802Z Running 0 items in this shard: 2025-12-04T09:31:22.0190031Z 2025-12-04T09:31:22.0192010Z Running distributed tests for the nccl backend with env init_method 2025-12-04T09:31:22.0193821Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:31:22.0198305Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:31:22.019611] 2025-12-04T09:32:11.1367962Z 2025-12-04T09:32:11.1369337Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_fb0b7b49049cc977_.log 2025-12-04T09:32:11.1374172Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2025-12-04T09:32:11.1378492Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2025-12-04T09:32:11.1379904Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2025-12-04T09:32:11.1381230Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2025-12-04T09:32:11.1382559Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2025-12-04T09:32:11.1383914Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2025-12-04T09:32:11.1385301Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2025-12-04T09:32:11.1386050Z 2025-12-04T09:32:11.1386321Z Running distributed tests for the nccl backend with file init_method 2025-12-04T09:32:11.1386815Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:32:11.1388888Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:32:11.137846] 2025-12-04T09:33:00.4088561Z 2025-12-04T09:33:00.4089899Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_4c14b89a54f988ad_.log 2025-12-04T09:33:00.4094648Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2025-12-04T09:33:00.4099230Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2025-12-04T09:33:00.4100657Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2025-12-04T09:33:00.4101994Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2025-12-04T09:33:00.4103307Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2025-12-04T09:33:00.4104672Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2025-12-04T09:33:00.4106062Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2025-12-04T09:33:00.4106818Z 2025-12-04T09:33:00.4107086Z Running distributed tests for the gloo backend with env init_method 2025-12-04T09:33:00.4107701Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:33:00.4109368Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:33:00.410002] 2025-12-04T09:33:32.7879992Z 2025-12-04T09:33:32.7881358Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_f309ef666470aa96_.log 2025-12-04T09:33:32.7886054Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2025-12-04T09:33:32.7890310Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2025-12-04T09:33:32.7891955Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2025-12-04T09:33:32.7893300Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2025-12-04T09:33:32.7894627Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2025-12-04T09:33:32.7895977Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2025-12-04T09:33:32.7897363Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2025-12-04T09:33:32.7898124Z 2025-12-04T09:33:32.7898388Z Running distributed tests for the gloo backend with file init_method 2025-12-04T09:33:32.7898963Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:33:32.7900396Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:33:32.789221] 2025-12-04T09:34:05.2663012Z 2025-12-04T09:34:05.2664461Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_cca15586252a4feb_.log 2025-12-04T09:34:05.2669973Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2025-12-04T09:34:05.2674418Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2025-12-04T09:34:05.2675791Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2025-12-04T09:34:05.2677172Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2025-12-04T09:34:05.2678544Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2025-12-04T09:34:05.2680049Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2025-12-04T09:34:05.2681432Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2025-12-04T09:34:05.2682199Z 2025-12-04T09:34:05.2682711Z Finished distributed/algorithms/quantization/test_quantization 1/1 ... [2025-12-04 09:34:05.266565][2476.204200283], took 2.97min 2025-12-04T09:34:05.2751029Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-daac7fd3db05a588.xml 2025-12-04T09:34:05.3349682Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-a07a6d6cccb031bb.xml 2025-12-04T09:34:05.3638195Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-7c0282eac181e46a.xml 2025-12-04T09:34:05.3900434Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-69522635455745bd.xml 2025-12-04T09:34:05.4113754Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-b1c2edd3b83dd6a9.xml 2025-12-04T09:34:05.4345438Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-8fc6dea04b4e72e8.xml 2025-12-04T09:34:05.4569658Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-694fabcdc25b908f.xml 2025-12-04T09:34:05.4851095Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cc8c47d77c37f668.xml 2025-12-04T09:34:05.5107375Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-93164f6c3fb62a8d.xml 2025-12-04T09:34:05.5427577Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-55976768b0d94051.xml 2025-12-04T09:34:05.5716137Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-ed34111b302d3808.xml 2025-12-04T09:34:05.6026731Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-160b76a599e6e82f.xml 2025-12-04T09:34:05.6338999Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-dbafcff6c2749b84.xml 2025-12-04T09:34:05.6648389Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-80abfc2e08c5c31a.xml 2025-12-04T09:34:05.6960518Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-281d630b2fcf32af.xml 2025-12-04T09:34:05.7240310Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-875849d617d7b51e.xml 2025-12-04T09:34:05.7500871Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-4cfe0d1200d89b83.xml 2025-12-04T09:34:05.7806863Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-366840efd0ab41ce.xml 2025-12-04T09:34:05.8086551Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-923db93bebc1ae0d.xml 2025-12-04T09:34:05.8334863Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cab701c96fd8d8eb.xml 2025-12-04T09:34:05.8858546Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-e799b1e80410ac4c.xml 2025-12-04T09:34:05.9138180Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-17c54a2ebe2f0c44.xml 2025-12-04T09:34:05.9417841Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-275ce76726773c8f.xml 2025-12-04T09:34:05.9717838Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-086dc66f538a7679.xml 2025-12-04T09:34:06.0177839Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-388ce163b11d3c23.xml 2025-12-04T09:34:06.0438850Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-e6a8c3241228684f.xml 2025-12-04T09:34:06.0718763Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cd4a02e6e5b27aca.xml 2025-12-04T09:34:06.0966749Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-c2167be978f3dc6a.xml 2025-12-04T09:34:06.1282610Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-14585f3cbdd695ec.xml 2025-12-04T09:34:06.1589913Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-bfadba8f7e8ab3cd.xml 2025-12-04T09:34:06.1890565Z Running distributed/test_c10d_object_collectives 1/1 ... [2025-12-04 09:34:06.188478][2477.126114215] 2025-12-04T09:34:06.1891234Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:34:06.1892533Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:34:06.188807] 2025-12-04T09:35:02.4504493Z 2025-12-04T09:35:02.4505665Z distributed/test_c10d_object_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_object_collectives_1.1_2145e99c0da21434_.log 2025-12-04T09:35:02.4511774Z Running 9 items in this shard: test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_broadcast_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_scatter_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_send_recv_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_broadcast_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_scatter_object 2025-12-04T09:35:02.4516480Z 2025-12-04T09:35:02.4516892Z Finished distributed/test_c10d_object_collectives 1/1 ... [2025-12-04 09:35:02.450057][2533.387691986], took 0.94min 2025-12-04T09:35:02.4587157Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_object_collectives/distributed.test_c10d_object_collectives-d97a03c8ce93a326.xml 2025-12-04T09:35:02.5474597Z Running distributed/tensor/debug/test_op_coverage 1/1 ... [2025-12-04 09:35:02.547222][2533.484857545] 2025-12-04T09:35:02.5475268Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:35:02.5478046Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:35:02.547580] 2025-12-04T09:35:06.8225740Z 2025-12-04T09:35:06.8226911Z distributed/tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.debug.test_op_coverage_1.1_0b0be77fb03d384c_.log 2025-12-04T09:35:06.8228707Z Running 1 items in this shard: test/distributed/tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2025-12-04T09:35:06.8229416Z 2025-12-04T09:35:06.8229866Z Finished distributed/tensor/debug/test_op_coverage 1/1 ... [2025-12-04 09:35:06.821969][2537.759602864], took 0.07min 2025-12-04T09:35:06.8306736Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.debug.test_op_coverage/distributed.tensor.debug.test_op_coverage-ebb808e01daa256c.xml 2025-12-04T09:35:06.8672449Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2025-12-04 09:35:06.867015][2537.804650847] 2025-12-04T09:35:06.8673172Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:35:06.8675891Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:35:06.867381] 2025-12-04T09:35:28.5842177Z 2025-12-04T09:35:28.5845181Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_1733371dd5ba4a20_.log 2025-12-04T09:35:28.5876944Z Running 44 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTP4GPUTest::test_extra_collectives 2025-12-04T09:35:28.5907627Z 2025-12-04T09:35:28.5908300Z Finished distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2025-12-04 09:35:28.583919][2559.521549498], took 0.36min 2025-12-04T09:35:28.5926877Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-3fe654a75937f34f.xml 2025-12-04T09:35:28.6884454Z Running distributed/_tools/test_mod_tracker 1/1 ... [2025-12-04 09:35:28.687855][2559.625491496] 2025-12-04T09:35:28.6885081Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:35:28.6886345Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:35:28.688191] 2025-12-04T09:35:32.4625409Z 2025-12-04T09:35:32.4626519Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_4c7520fa7d7771cc_.log 2025-12-04T09:35:32.4629184Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2025-12-04T09:35:32.4630891Z 2025-12-04T09:35:32.4631307Z Finished distributed/_tools/test_mod_tracker 1/1 ... [2025-12-04 09:35:32.462029][2563.399660285], took 0.06min 2025-12-04T09:35:32.4709588Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-2a8ca78a6e78af3b.xml 2025-12-04T09:35:32.5126330Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2025-12-04 09:35:32.512155][2563.449793055] 2025-12-04T09:35:32.5127045Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:35:32.5128415Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:35:32.512532] 2025-12-04T09:35:36.2867588Z 2025-12-04T09:35:36.2869311Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_b7e7178e36b6771f_.log 2025-12-04T09:35:36.2871008Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2025-12-04T09:35:36.2871720Z 2025-12-04T09:35:36.2872173Z Finished distributed/_shard/sharded_tensor/test_logger 1/1 ... [2025-12-04 09:35:36.286182][2567.223817162], took 0.06min 2025-12-04T09:35:36.2953443Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-8d04d60499ed13c6.xml 2025-12-04T09:35:36.3327228Z Running distributed/tensor/test_dtensor_compile 1/1 ... [2025-12-04 09:35:36.332443][2567.270079319] 2025-12-04T09:35:36.3327888Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:35:36.3329803Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:35:36.332791] 2025-12-04T09:39:59.9088212Z 2025-12-04T09:39:59.9089380Z distributed/tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_compile_1.1_756cc2654f1e6bbb_.log 2025-12-04T09:39:59.9118198Z Running 49 items in this shard: test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_basic, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_basic_export, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_cat, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_loss_parallel_log_softmax, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_recompiles, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_slice, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_requires_grad_recompile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_async, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_from_local_grad_placements_sequence_intermediate, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_from_local_grad_placements_sequence_intermediate_as_args, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_grad_placements_sequence, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_grad_placements_sequence_intermediate, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_get_local_rank_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering_graph_partition, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_embedding_redistribute, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_ca_True 2025-12-04T09:39:59.9145992Z 2025-12-04T09:39:59.9146419Z Finished distributed/tensor/test_dtensor_compile 1/1 ... [2025-12-04 09:39:59.908445][2830.846080278], took 4.39min 2025-12-04T09:39:59.9176790Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_dtensor_compile/distributed.tensor.test_dtensor_compile-b72028e76a0d9d1d.xml 2025-12-04T09:39:59.9969390Z Running distributed/test_aten_comm_compute_reordering 2/2 ... [2025-12-04 09:39:59.996371][2830.934008327] 2025-12-04T09:39:59.9970067Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:39:59.9971393Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_aten_comm_compute_reordering.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:39:59.996706] 2025-12-04T09:45:32.5829721Z 2025-12-04T09:45:32.5830909Z distributed/test_aten_comm_compute_reordering 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_aten_comm_compute_reordering_2.2_292d0b2b786a1255_.log 2025-12-04T09:45:32.5847093Z Running 23 items in this shard: test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingMultiProc::test_custom_estimator_for_non_compute_nodes, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingMultiProc::test_inductor_default_comms_ordering, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingMultiProc::test_overlap_scheduling_via_config, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_mul, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingMultiProc::test_schedulable_wait, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_basic_all_gather_bucketing, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_bucketing_with_convert_dtype, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_collective_benchmarking_with_real_pg, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_custom_estimation_with_fake_tensor_mode, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_custom_estimator_for_non_compute_nodes, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_grouped_scheduler_node, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_multiple_hiding_nodes_bucketing, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_no_bucketing_with_dependent_hiding_nodes, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_raise_comms, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_reduce_scatter_bucketing, test/distributed/test_aten_comm_compute_reordering.py::TestComputeCommReorderingBucketing::test_sink_waits_raise_comms, test/distributed/test_aten_comm_compute_reordering.py::TestManualOverlapBucketing::test_bucketing_reordering_pass_single_bucket_custom_module_stack_fn, test/distributed/test_aten_comm_compute_reordering.py::TestManualOverlapBucketing::test_custom_estimator_for_non_compute_nodes, test/distributed/test_aten_comm_compute_reordering.py::TestManualOverlapBucketing::test_inductor_default_comms_ordering, test/distributed/test_aten_comm_compute_reordering.py::TestManualOverlapBucketing::test_make_graph_view_and_get_subgraph_by_path, test/distributed/test_aten_comm_compute_reordering.py::TestManualOverlapBucketing::test_overlap_scheduling_via_config, test/distributed/test_aten_comm_compute_reordering.py::TestManualOverlapBucketing::test_raise_comms 2025-12-04T09:45:32.5863314Z 2025-12-04T09:45:32.5863797Z Finished distributed/test_aten_comm_compute_reordering 2/2 ... [2025-12-04 09:45:32.583024][3163.520659398], took 5.54min 2025-12-04T09:45:32.5924458Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_aten_comm_compute_reordering/distributed.test_aten_comm_compute_reordering-7d1c62987df9c762.xml 2025-12-04T09:45:32.9543728Z Uploading artifacts took 0.27 seconds 2025-12-04T09:45:32.9547333Z Running distributed/_tools/test_sac_estimator 1/1 ... [2025-12-04 09:45:32.954447][3163.892084327] 2025-12-04T09:45:32.9548149Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:45:32.9549938Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:45:32.954790] 2025-12-04T09:45:40.2358854Z 2025-12-04T09:45:40.2360414Z distributed/_tools/test_sac_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_estimator_1.1_687d9b4d32abbd61_.log 2025-12-04T09:45:40.2362461Z Running 2 items in this shard: test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_simple_model_sac_estimation, test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_transformer_sac_estimation 2025-12-04T09:45:40.2363595Z 2025-12-04T09:45:40.2364005Z Finished distributed/_tools/test_sac_estimator 1/1 ... [2025-12-04 09:45:40.235462][3171.173098074], took 0.12min 2025-12-04T09:45:40.2446152Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._tools.test_sac_estimator/distributed._tools.test_sac_estimator-b0bd4af0f86e3ece.xml 2025-12-04T09:45:40.3210499Z Running distributed/_tools/test_memory_tracker 1/1 ... [2025-12-04 09:45:40.320545][3171.258182542] 2025-12-04T09:45:40.3211225Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:45:40.3212697Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:45:40.320877] 2025-12-04T09:45:44.7970255Z 2025-12-04T09:45:44.7971439Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_d73934ab53527d1b_.log 2025-12-04T09:45:44.7973259Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2025-12-04T09:45:44.7973835Z 2025-12-04T09:45:44.7974248Z Finished distributed/_tools/test_memory_tracker 1/1 ... [2025-12-04 09:45:44.796529][3175.734160071], took 0.07min 2025-12-04T09:45:44.8060261Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-c7b456a4f8311d2d.xml 2025-12-04T09:45:44.8522593Z Running distributed/checkpoint/_experimental/test_builder 1/1 ... [2025-12-04 09:45:44.851697][3175.789334915] 2025-12-04T09:45:44.8523318Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:45:44.8525099Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_builder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:45:44.852034] 2025-12-04T09:45:52.4351138Z 2025-12-04T09:45:52.4352558Z distributed/checkpoint/_experimental/test_builder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_builder_1.1_08d5e7d49c7ad5da_.log 2025-12-04T09:45:52.4356289Z Running 4 items in this shard: test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_async_checkpointer, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer_with_config_first, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer_with_custom_config 2025-12-04T09:45:52.4358967Z 2025-12-04T09:45:52.4359436Z Finished distributed/checkpoint/_experimental/test_builder 1/1 ... [2025-12-04 09:45:52.434924][3183.372560626], took 0.13min 2025-12-04T09:45:52.4447072Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint._experimental.test_builder/distributed.checkpoint._experimental.test_builder-63eb721b9e0f0e0e.xml 2025-12-04T09:45:52.5322174Z Running distributed/_composable/test_replicate_with_fsdp 1/1 ... [2025-12-04 09:45:52.531661][3183.469298071] 2025-12-04T09:45:52.5322879Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:45:52.5324904Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate_with_fsdp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:45:52.531995] 2025-12-04T09:46:27.9866616Z 2025-12-04T09:46:27.9868198Z distributed/_composable/test_replicate_with_fsdp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_fsdp_1.1_8cf393c82285846c_.log 2025-12-04T09:46:27.9872344Z Running 5 items in this shard: test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_tp_device_mesh, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_transformer, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_transformer_managed_modules, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_train_parity_2d_mlp, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_train_replicate_fsdp 2025-12-04T09:46:27.9875423Z 2025-12-04T09:46:27.9875895Z Finished distributed/_composable/test_replicate_with_fsdp 1/1 ... [2025-12-04 09:46:27.986247][3218.923882887], took 0.59min 2025-12-04T09:46:27.9965646Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._composable.test_replicate_with_fsdp/distributed._composable.test_replicate_with_fsdp-4c94f165a9954ceb.xml 2025-12-04T09:46:28.0651495Z Running distributed/tensor/test_xla_integration 1/1 ... [2025-12-04 09:46:28.064642][3219.002279274] 2025-12-04T09:46:28.0652156Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:46:28.0653451Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:46:28.064965] 2025-12-04T09:46:31.8895018Z 2025-12-04T09:46:31.8896155Z distributed/tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_xla_integration_1.1_d107779fabf62ae7_.log 2025-12-04T09:46:31.8898882Z Running 3 items in this shard: test/distributed/tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2025-12-04T09:46:31.8900753Z 2025-12-04T09:46:31.8901175Z Finished distributed/tensor/test_xla_integration 1/1 ... [2025-12-04 09:46:31.888898][3222.826532991], took 0.06min 2025-12-04T09:46:31.8989590Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_xla_integration/distributed.tensor.test_xla_integration-953545bf454723f1.xml 2025-12-04T09:46:31.9356028Z Running distributed/checkpoint/_experimental/test_types 1/1 ... [2025-12-04 09:46:31.935376][3222.873013138] 2025-12-04T09:46:31.9356734Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:46:31.9359000Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_types.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:46:31.935697] 2025-12-04T09:46:35.8099031Z 2025-12-04T09:46:35.8100307Z distributed/checkpoint/_experimental/test_types 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_types_1.1_729af79fa7fcfdd8_.log 2025-12-04T09:46:35.8103344Z Running 3 items in this shard: test/distributed/checkpoint/_experimental/test_types.py::TestRankInfo::test_rank_info_default_initialization, test/distributed/checkpoint/_experimental/test_types.py::TestRankInfo::test_rank_info_initialization, test/distributed/checkpoint/_experimental/test_types.py::TestRankInfo::test_state_dict_type_alias 2025-12-04T09:46:35.8105116Z 2025-12-04T09:46:35.8105590Z Finished distributed/checkpoint/_experimental/test_types 1/1 ... [2025-12-04 09:46:35.809301][3226.746936512], took 0.06min 2025-12-04T09:46:35.8190785Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint._experimental.test_types/distributed.checkpoint._experimental.test_types-02c661f341e1f68e.xml 2025-12-04T09:46:35.8568816Z Running distributed/tensor/experimental/test_register_sharding 1/1 ... [2025-12-04 09:46:35.856265][3226.793902053] 2025-12-04T09:46:35.8569565Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:46:35.8570963Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:46:35.856603] 2025-12-04T09:46:57.7266439Z 2025-12-04T09:46:57.7268151Z distributed/tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.experimental.test_register_sharding_1.1_043cb655dab57eee_.log 2025-12-04T09:46:57.7271260Z Running 3 items in this shard: test/distributed/tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_register_sharding_for_tensor_kwargs, test/distributed/tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd 2025-12-04T09:46:57.7273404Z 2025-12-04T09:46:57.7273937Z Finished distributed/tensor/experimental/test_register_sharding 1/1 ... [2025-12-04 09:46:57.726060][3248.663696247], took 0.36min 2025-12-04T09:46:57.7364087Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.experimental.test_register_sharding/distributed.tensor.experimental.test_register_sharding-ee24555d8c546070.xml 2025-12-04T09:46:57.8248262Z Running distributed/test_backends 1/1 ... [2025-12-04 09:46:57.824201][3248.761838109] 2025-12-04T09:46:57.8248918Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:46:57.8250144Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_backends.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:46:57.824536] 2025-12-04T09:47:02.9023391Z 2025-12-04T09:47:02.9024958Z distributed/test_backends 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_backends_1.1_75335fea59b22df5_.log 2025-12-04T09:47:02.9026894Z Running 2 items in this shard: test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_create_pg_cuda, test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_device_to_backend_mapping_cuda 2025-12-04T09:47:02.9028144Z 2025-12-04T09:47:02.9028504Z Finished distributed/test_backends 1/1 ... [2025-12-04 09:47:02.901725][3253.839360215], took 0.08min 2025-12-04T09:47:02.9119369Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_backends/distributed.test_backends-11b04c9998663597.xml 2025-12-04T09:47:02.9457409Z Running distributed/tensor/test_experimental_ops 1/1 ... [2025-12-04 09:47:02.945138][3253.882775037] 2025-12-04T09:47:02.9458068Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:47:02.9459370Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:47:02.945478] 2025-12-04T09:47:23.0596070Z 2025-12-04T09:47:23.0597553Z distributed/tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_experimental_ops_1.1_19a129c7d3ded4d3_.log 2025-12-04T09:47:23.0601493Z Running 6 items in this shard: test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTestWithLocalTensor::test_bernoulli, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTestWithLocalTensor::test_nll, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTestWithLocalTensor::test_slice 2025-12-04T09:47:23.0604399Z 2025-12-04T09:47:23.0604830Z Finished distributed/tensor/test_experimental_ops 1/1 ... [2025-12-04 09:47:23.059387][3273.997022602], took 0.34min 2025-12-04T09:47:23.0703463Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_experimental_ops/distributed.tensor.test_experimental_ops-1d5fbe2b841d7120.xml 2025-12-04T09:47:23.1494861Z Running distributed/checkpoint/test_quantized_hf_storage 1/1 ... [2025-12-04 09:47:23.148863][3274.086501115] 2025-12-04T09:47:23.1495571Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:47:23.1496903Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_quantized_hf_storage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:47:23.149197] 2025-12-04T09:47:27.0237754Z 2025-12-04T09:47:27.0239074Z distributed/checkpoint/test_quantized_hf_storage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_quantized_hf_storage_1.1_e66d1a41faed9c49_.log 2025-12-04T09:47:27.0241467Z Running 2 items in this shard: test/distributed/checkpoint/test_quantized_hf_storage.py::TestQuantizedHfStorage::test_dequantization, test/distributed/checkpoint/test_quantized_hf_storage.py::TestQuantizedHfStorage::test_dtensor_slice_dequantization_block_alignment 2025-12-04T09:47:27.0242967Z 2025-12-04T09:47:27.0243448Z Finished distributed/checkpoint/test_quantized_hf_storage 1/1 ... [2025-12-04 09:47:27.023226][3277.960857275], took 0.06min 2025-12-04T09:47:27.0340499Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_quantized_hf_storage/distributed.checkpoint.test_quantized_hf_storage-b35612bb520e65c5.xml 2025-12-04T09:47:27.0686141Z Running distributed/_composable/test_composability/test_pp_composability 1/1 ... [2025-12-04 09:47:27.067995][3278.005632402] 2025-12-04T09:47:27.0686953Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:47:27.0688396Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_pp_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:47:27.068324] 2025-12-04T09:47:30.5913619Z 2025-12-04T09:47:30.5915056Z distributed/_composable/test_composability/test_pp_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_pp_composability_1.1_626d47024cd05ebf_.log 2025-12-04T09:47:30.5934914Z Running 26 items in this shard: test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass0_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass0_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass1_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass1_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass2_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass2_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass3_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass3_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass4_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass4_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_pp_and_dcp, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass0_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass0_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass1_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass1_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass2_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass2_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass3_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass3_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass4_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_ScheduleClass4_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_grads_ScheduleClass0, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_grads_ScheduleClass1, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_grads_ScheduleClass2, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_grads_ScheduleClass3, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_replicate_pp_grads_ScheduleClass4 2025-12-04T09:47:30.5952986Z 2025-12-04T09:47:30.5953657Z Finished distributed/_composable/test_composability/test_pp_composability 1/1 ... [2025-12-04 09:47:30.591172][3281.5288083], took 0.06min 2025-12-04T09:47:30.6021016Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._composable.test_composability.test_pp_composability/distributed._composable.test_composability.test_pp_composability-6aa49750a7598206.xml 2025-12-04T09:47:30.6323957Z Running distributed/checkpoint/test_async_process_executor 1/1 ... [2025-12-04 09:47:30.631698][3281.56933502] 2025-12-04T09:47:30.6324863Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:47:30.6326482Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_async_process_executor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:47:30.632025] 2025-12-04T09:48:08.3926397Z 2025-12-04T09:48:08.3927683Z distributed/checkpoint/test_async_process_executor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_async_process_executor_1.1_510a90a3345e7d90_.log 2025-12-04T09:48:08.3932475Z Running 5 items in this shard: test/distributed/checkpoint/test_async_process_executor.py::TestAsyncProcessExecutor::test_checkpoint_save_failure_continues_serving, test/distributed/checkpoint/test_async_process_executor.py::TestAsyncProcessExecutorPrefixStore::test_checkpoint_save_with_prefix_store_enabled, test/distributed/checkpoint/test_async_process_executor.py::TestProcessGroupInitInfo::test_process_group_init_info_with_default_pg, test/distributed/checkpoint/test_async_process_executor.py::TestProcessGroupInitInfo::test_process_group_init_info_with_prefix_store_env_var, test/distributed/checkpoint/test_async_process_executor.py::TestProcessGroupInitInfo::test_process_group_init_info_without_prefix_store_env_var 2025-12-04T09:48:08.3936604Z 2025-12-04T09:48:08.3937099Z Finished distributed/checkpoint/test_async_process_executor 1/1 ... [2025-12-04 09:48:08.392517][3319.33015108], took 0.63min 2025-12-04T09:48:08.4032058Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_async_process_executor/distributed.checkpoint.test_async_process_executor-257bf63a3e72d6aa.xml 2025-12-04T09:48:08.4968222Z Running distributed/tensor/test_tensor_ops 1/1 ... [2025-12-04 09:48:08.496225][3319.433862573] 2025-12-04T09:48:08.4968850Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:48:08.4970111Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:48:08.496557] 2025-12-04T09:53:32.1671523Z 2025-12-04T09:53:32.1672644Z distributed/tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_tensor_ops_1.1_cb32e25d0c49245f_.log 2025-12-04T09:53:32.1701984Z Running 62 items in this shard: test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_copy_, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_index_put_scalar, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_index_put_tensor, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_slice, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_split_on_partial, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack_cache, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_unbind, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_aten_contiguous, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_clone, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_contiguous, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_copy_, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_detach, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_dtensor_dtype_conversion, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_empty_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_equal, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_fill_inplace, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_fill_inplace_partial_sum, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_full_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_gather, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_index, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_index_put_scalar, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_index_put_tensor, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_inplace_op, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_new_empty_strided, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_new_full, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_ones_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_ones_like_partial_sum, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_op_out_variant, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_scatter, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_slice, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_split_on_partial, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_stack, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_stack_cache, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_unbind, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_where_type_promotion, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_zero_inplace, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_zeros_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTestWithLocalTensor::test_zeros_like_partial_sum 2025-12-04T09:53:32.1731485Z 2025-12-04T09:53:32.1731907Z Finished distributed/tensor/test_tensor_ops 1/1 ... [2025-12-04 09:53:32.167005][3643.104639152], took 5.39min 2025-12-04T09:53:32.1785446Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_tensor_ops/distributed.tensor.test_tensor_ops-dd61e7522cc11d1f.xml 2025-12-04T09:53:32.2749159Z Running distributed/test_device_mesh 1/1 ... [2025-12-04 09:53:32.274262][3643.211899338] 2025-12-04T09:53:32.2749783Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:53:32.2751047Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:53:32.274596] 2025-12-04T09:58:52.6103555Z 2025-12-04T09:58:52.6104594Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_155d8072915be213_.log 2025-12-04T09:58:52.6134992Z Running 66 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTestGlooBackend::test_device_mesh_reuse_default_group, test/distributed/test_device_mesh.py::DeviceMeshSetDeviceTest::test_auto_set_device_from_heuristic, test/distributed/test_device_mesh.py::DeviceMeshSetDeviceTest::test_auto_set_device_from_local_rank, test/distributed/test_device_mesh.py::DeviceMeshSetDeviceTest::test_manual_set_device, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_non_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_init_backend, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_global_pg, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_invalid_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_and_get_all_groups, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_root_mesh_multiple_independent_meshes, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_raises_invalid_device_type, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_parent_child_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape_2d, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape_3d, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_get_local_rank_3d, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_dict_with_idx_and_backend_eager, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_dict_with_idx_and_backend_lazy, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_dict_with_name_and_options, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_errors, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_concatenate_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_concatenate_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_4d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d_noncontiguous_slicing, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_reconstruct_mesh_with_flatten_dim, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_unflatten_mesh_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_unflatten_mesh_3d, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_all_submeshes, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_not_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_mesh_slice_fake_tensor_mode, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven, test/distributed/test_device_mesh.py::CuTeLayoutTest::test_check_non_overlap, test/distributed/test_device_mesh.py::CuTeLayoutTest::test_coalesce, test/distributed/test_device_mesh.py::CuTeLayoutTest::test_coalesce_non_coalescible, test/distributed/test_device_mesh.py::CuTeLayoutTest::test_complement_n_group_layout, test/distributed/test_device_mesh.py::CuTeLayoutTest::test_composition, test/distributed/test_device_mesh.py::CuTeLayoutTest::test_remap_to_tensor 2025-12-04T09:58:52.6162791Z 2025-12-04T09:58:52.6163137Z Finished distributed/test_device_mesh 1/1 ... [2025-12-04 09:58:52.611133][3963.54876613], took 5.34min 2025-12-04T09:58:52.6226155Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-2d037d076dad7e9e.xml 2025-12-04T09:58:52.7010290Z Running distributed/optim/test_named_optimizer 1/1 ... [2025-12-04 09:58:52.700352][3963.637989427] 2025-12-04T09:58:52.7010975Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:58:52.7013444Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:58:52.700684] 2025-12-04T09:58:55.0929349Z 2025-12-04T09:58:55.0930522Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_72bde93862166bdc_.log 2025-12-04T09:58:55.0931562Z 2025-12-04T09:58:55.0931993Z Finished distributed/optim/test_named_optimizer 1/1 ... [2025-12-04 09:58:55.092730][3966.030366215], took 0.04min 2025-12-04T09:58:55.1073532Z Running distributed/_composable/fsdp/test_fully_shard_ignore_params 1/1 ... [2025-12-04 09:58:55.107107][3966.044744575] 2025-12-04T09:58:55.1074289Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:58:55.1076787Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_ignore_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:58:55.107451] 2025-12-04T09:59:04.7968459Z 2025-12-04T09:59:04.7969792Z distributed/_composable/fsdp/test_fully_shard_ignore_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_ignore_params_1.1_5b98885cf0fe1d50_.log 2025-12-04T09:59:04.7971654Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_ignore_params.py::TestFullyShardIgnoreParams::test_ddp_A_fsdp_B_ddp_C 2025-12-04T09:59:04.7972419Z 2025-12-04T09:59:04.7972945Z Finished distributed/_composable/fsdp/test_fully_shard_ignore_params 1/1 ... [2025-12-04 09:59:04.796225][3975.733862316], took 0.16min 2025-12-04T09:59:04.8073780Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_ignore_params/distributed._composable.fsdp.test_fully_shard_ignore_params-3bd5835b23ec8132.xml 2025-12-04T09:59:04.8909614Z Running distributed/checkpoint/_experimental/test_staging 1/1 ... [2025-12-04 09:59:04.890394][3975.828030871] 2025-12-04T09:59:04.8910353Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:59:04.8911748Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_staging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:59:04.890745] 2025-12-04T09:59:09.1663072Z 2025-12-04T09:59:09.1664239Z distributed/checkpoint/_experimental/test_staging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_staging_1.1_3d837f3dbe22921b_.log 2025-12-04T09:59:09.1669501Z Running 7 items in this shard: test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_async_staging, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_cuda_non_blocking_without_cuda, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_cuda_tensors_staging, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_different_option_combinations, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_multiple_staging_operations, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_resource_cleanup, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_sync_staging 2025-12-04T09:59:09.1673753Z 2025-12-04T09:59:09.1674208Z Finished distributed/checkpoint/_experimental/test_staging 1/1 ... [2025-12-04 09:59:09.165768][3980.103404156], took 0.07min 2025-12-04T09:59:09.1774634Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint._experimental.test_staging/distributed.checkpoint._experimental.test_staging-07a6a2de828c3109.xml 2025-12-04T09:59:09.2134762Z Running distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2025-12-04 09:59:09.213247][3980.150882718] 2025-12-04T09:59:09.2135530Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:59:09.2137571Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:59:09.213564] 2025-12-04T09:59:19.3044296Z 2025-12-04T09:59:19.3045720Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_16c4b7a17248643a_.log 2025-12-04T09:59:19.3047955Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py::TestFsdpTpCheckpointConversion::test_fsdp_to_tp 2025-12-04T09:59:19.3048802Z 2025-12-04T09:59:19.3049331Z Finished distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2025-12-04 09:59:19.303821][3990.241457321], took 0.17min 2025-12-04T09:59:19.3151575Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion-7932544576c43c55.xml 2025-12-04T09:59:19.3974727Z Running distributed/tensor/test_embedding_ops 1/1 ... [2025-12-04 09:59:19.396855][3990.334493032] 2025-12-04T09:59:19.3975368Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:59:19.3976677Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_embedding_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:59:19.397191] 2025-12-04T09:59:49.7389197Z 2025-12-04T09:59:49.7390349Z distributed/tensor/test_embedding_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_embedding_ops_1.1_bb5002b50196bdaf_.log 2025-12-04T09:59:49.7395865Z Running 8 items in this shard: test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOp::test_multiple_embeddings_rowwise, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise_max_norm_errors, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_rowwise, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOpWithLocalTensor::test_multiple_embeddings_rowwise, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOpWithLocalTensor::test_sharded_embedding_colwise, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOpWithLocalTensor::test_sharded_embedding_colwise_max_norm_errors, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOpWithLocalTensor::test_sharded_embedding_rowwise 2025-12-04T09:59:49.7400973Z 2025-12-04T09:59:49.7401385Z Finished distributed/tensor/test_embedding_ops 1/1 ... [2025-12-04 09:59:49.738484][4020.676120409], took 0.51min 2025-12-04T09:59:49.7508710Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_embedding_ops/distributed.tensor.test_embedding_ops-389c4439b22d140b.xml 2025-12-04T09:59:49.8396898Z Running distributed/tensor/experimental/test_local_map 1/1 ... [2025-12-04 09:59:49.839443][4020.777079287] 2025-12-04T09:59:49.8397629Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T09:59:49.8400141Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/experimental/test_local_map.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 09:59:49.839809] 2025-12-04T10:00:24.2431041Z 2025-12-04T10:00:24.2432433Z distributed/tensor/experimental/test_local_map 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.experimental.test_local_map_1.1_25f283957a7af0d4_.log 2025-12-04T10:00:24.2437038Z Running 6 items in this shard: test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_correctness, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_in_placements, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_out_placements, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_redistribute, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_with_grad_placement, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_multi_mesh_inputs 2025-12-04T10:00:24.2440283Z 2025-12-04T10:00:24.2548962Z Finished distributed/tensor/experimental/test_local_map 1/1 ... [2025-12-04 10:00:24.242939][4055.180575357], took 0.57min 2025-12-04T10:00:24.2550651Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.experimental.test_local_map/distributed.tensor.experimental.test_local_map-43637351f823edb4.xml 2025-12-04T10:00:24.3491094Z Running distributed/test_local_tensor 1/1 ... [2025-12-04 10:00:24.348526][4055.286163497] 2025-12-04T10:00:24.3491703Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:00:24.3492944Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_local_tensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:00:24.348865] 2025-12-04T10:00:29.2756308Z 2025-12-04T10:00:29.2758637Z distributed/test_local_tensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_local_tensor_1.1_2cdd9cdded794bb9_.log 2025-12-04T10:00:29.2769850Z Running 21 items in this shard: test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_basic_arithmetic_operations, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_collectives_within_local_tensor_mode, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_empty_local_tensors, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_even_sharding_mean_is_partial, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_local_tensor_creation_fails_with_grad_tensors, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_local_tensor_dtype_consistency, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_local_tensor_mode, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_mixed_operations_with_regular_tensors, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_scalar_mul_reduction_bug, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_uneven_sharding_mean_bug, test/distributed/test_local_tensor.py::TestLocalTensorWorld2::test_uneven_sharding_prod, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_all_gather_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_all_gather_into_tensor_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_all_reduce_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_all_to_all_single_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_broadcast_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_collective_reduction_operations, test/distributed/test_local_tensor.py::TestLocalTensorWorld3::test_reduce_scatter_tensor_collective, test/distributed/test_local_tensor.py::TestLocalTensorWorld4::test_dtensor_cat, test/distributed/test_local_tensor.py::TestLocalTensorWorld8::test_dtensor_addmm, test/distributed/test_local_tensor.py::TestLocalRunner::test_dp_pp 2025-12-04T10:00:29.2779906Z 2025-12-04T10:00:29.2780261Z Finished distributed/test_local_tensor 1/1 ... [2025-12-04 10:00:29.275325][4060.212956216], took 0.08min 2025-12-04T10:00:29.2875759Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_local_tensor/distributed.test_local_tensor-2d16d75957767ef5.xml 2025-12-04T10:00:29.3326586Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2025-12-04 10:00:29.332276][4060.269913114] 2025-12-04T10:00:29.3327447Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:00:29.3328855Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:00:29.332643] 2025-12-04T10:00:33.5080464Z 2025-12-04T10:00:33.5081789Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_a3025a0aeaaae488_.log 2025-12-04T10:00:33.5085550Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2025-12-04T10:00:33.5088439Z 2025-12-04T10:00:33.5088978Z Finished distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2025-12-04 10:00:33.507550][4064.445180916], took 0.07min 2025-12-04T10:00:33.5194778Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-f6baa7b934e7caab.xml 2025-12-04T10:00:33.5711323Z Running distributed/checkpoint/test_tp_checkpoint 1/1 ... [2025-12-04 10:00:33.570906][4064.508543116] 2025-12-04T10:00:33.5712169Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:00:33.5714345Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_tp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:00:33.571237] 2025-12-04T10:00:49.8776494Z 2025-12-04T10:00:49.8777698Z distributed/checkpoint/test_tp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_0e886b22c4d1b3fa_.log 2025-12-04T10:00:49.8779745Z Running 2 items in this shard: test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint, test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint_load_on_meta_device 2025-12-04T10:00:49.8780929Z 2025-12-04T10:00:49.8781370Z Finished distributed/checkpoint/test_tp_checkpoint 1/1 ... [2025-12-04 10:00:49.877178][4080.814814213], took 0.27min 2025-12-04T10:00:49.8898229Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_tp_checkpoint/distributed.checkpoint.test_tp_checkpoint-11d7a0d46082f98d.xml 2025-12-04T10:00:49.9793988Z Running distributed/pipelining/test_stage 1/1 ... [2025-12-04 10:00:49.979161][4080.916797777] 2025-12-04T10:00:49.9794622Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:00:49.9796943Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_stage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:00:49.979502] 2025-12-04T10:01:13.6036610Z 2025-12-04T10:01:13.6037690Z distributed/pipelining/test_stage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_stage_1.1_ed32aa9322f8b97b_.log 2025-12-04T10:01:13.6042285Z Running 8 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_output_chunks_memory_usage, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0, test/distributed/pipelining/test_stage.py::StageNegativeTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageNegativeTest::test_shape_prop_mismatch 2025-12-04T10:01:13.6045697Z 2025-12-04T10:01:13.6046099Z Finished distributed/pipelining/test_stage 1/1 ... [2025-12-04 10:01:13.603191][4104.540827165], took 0.39min 2025-12-04T10:01:13.6160815Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-4e9f4b21d1bcf8f6.xml 2025-12-04T10:01:13.7259105Z Running distributed/tensor/parallel/test_tp_random_state 1/1 ... [2025-12-04 10:01:13.725350][4104.662987458] 2025-12-04T10:01:13.7259813Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:01:13.7261167Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_random_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:01:13.725696] 2025-12-04T10:01:23.9162020Z 2025-12-04T10:01:23.9163292Z distributed/tensor/parallel/test_tp_random_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_763ea794449f404f_.log 2025-12-04T10:01:23.9164982Z Running 1 items in this shard: test/distributed/tensor/parallel/test_tp_random_state.py::TensorParallelRandomStateTests::test_model_init 2025-12-04T10:01:23.9166098Z 2025-12-04T10:01:23.9166568Z Finished distributed/tensor/parallel/test_tp_random_state 1/1 ... [2025-12-04 10:01:23.915715][4114.853350961], took 0.17min 2025-12-04T10:01:23.9280473Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_random_state/distributed.tensor.parallel.test_tp_random_state-3d1a164d85d827e3.xml 2025-12-04T10:01:24.0227121Z Running distributed/checkpoint/test_planner 1/1 ... [2025-12-04 10:01:24.022350][4114.959987517] 2025-12-04T10:01:24.0227922Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:01:24.0230514Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:01:24.022849] 2025-12-04T10:01:28.3483424Z 2025-12-04T10:01:28.3484562Z distributed/checkpoint/test_planner 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_planner_1.1_44287fa663f3fa73_.log 2025-12-04T10:01:28.3494979Z Running 17 items in this shard: test/distributed/checkpoint/test_planner.py::TestSavePlan::test_dedup_plans, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_finish_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_resharding, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_world_size_diff_by_one, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_load_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_compare_save_plans, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_create_read_item_from_chunks, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_merge_delta_local_plans, test/distributed/checkpoint/test_planner.py::TestValidateGlobalPlan::test_detect_overlapping_chunks, test/distributed/checkpoint/test_planner.py::TestValidateGlobalPlan::test_non_overlapping_chunks, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_load_different_sizes_throws, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_strict, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_version_key_in_planner_data 2025-12-04T10:01:28.3502376Z 2025-12-04T10:01:28.3502754Z Finished distributed/checkpoint/test_planner 1/1 ... [2025-12-04 10:01:28.347860][4119.285495181], took 0.07min 2025-12-04T10:01:28.3604187Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_planner/distributed.checkpoint.test_planner-6c9d97ee371ac2d1.xml 2025-12-04T10:01:28.3971603Z Running distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2025-12-04 10:01:28.396622][4119.334259926] 2025-12-04T10:01:28.3972317Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:01:28.3973665Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:01:28.396957] 2025-12-04T10:01:38.4860341Z 2025-12-04T10:01:38.4861642Z distributed/checkpoint/test_dtensor_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_365b28a8675e628f_.log 2025-12-04T10:01:38.4863332Z Running 1 items in this shard: test/distributed/checkpoint/test_dtensor_checkpoint.py::DTensorPlanner::test_distributed_tensor_planner 2025-12-04T10:01:38.4864376Z 2025-12-04T10:01:38.4864843Z Finished distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2025-12-04 10:01:38.485497][4129.423127922], took 0.17min 2025-12-04T10:01:38.4984723Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_checkpoint/distributed.checkpoint.test_dtensor_checkpoint-cb541eaa40da104f.xml 2025-12-04T10:01:38.6011886Z Running distributed/pipelining/test_schedule 1/1 ... [2025-12-04 10:01:38.600579][4129.538215905] 2025-12-04T10:01:38.6012602Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:01:38.6013885Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:01:38.600918] 2025-12-04T10:02:38.4507824Z 2025-12-04T10:02:38.4509676Z distributed/pipelining/test_schedule 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_1.1_4fb1c6d0d88f6090_.log 2025-12-04T10:02:38.4535105Z Running 43 items in this shard: test/distributed/pipelining/test_schedule.py::ScheduleTest::test_get_schedule_class, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_eval_then_train_ScheduleClass0, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_eval_then_train_ScheduleClass1, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_eval_then_train_ScheduleClass2, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_eval_then_train_ScheduleClass3, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_eval_then_train_ScheduleClass4, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass0, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass1, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass2, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass3, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass4, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_zero_bubble_schedule_errors_with_compile_ScheduleClass0, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_zero_bubble_schedule_errors_with_compile_ScheduleClass1, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_zero_bubble_schedule_errors_with_compile_ScheduleClass2, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_flex_and_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_flex_and_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_for_v_schedules_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_for_v_schedules_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestScheduleCsv::test_csv_compare_ScheduleClass0_csv_name_dualpipev_4rank_10mb, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref2, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref3, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref4, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref5, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref6, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref7, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_csv_csv_name_zb1p_2rank_2stagep, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_grad_with_split_b_w, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_grad_with_v_schedule, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_merge_bw_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_reduce_grad_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_reduce_grad_test_info1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_send_recv_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_send_recv_test_info1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_unshard_reshard_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_unshard_reshard_test_info1, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_invalid_schedule_missing_action, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_invalid_schedule_missing_rank, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_valid_schedule, test/distributed/pipelining/test_schedule.py::ScheduleUtilTests::test_generate_stage_to_rank_mapping 2025-12-04T10:02:38.4558694Z 2025-12-04T10:02:38.4559123Z Finished distributed/pipelining/test_schedule 1/1 ... [2025-12-04 10:02:38.450263][4189.387893062], took 1.00min 2025-12-04T10:02:38.4638060Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.pipelining.test_schedule/distributed.pipelining.test_schedule-87ecb28af2fb8a27.xml 2025-12-04T10:02:38.5500958Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2025-12-04 10:02:38.549485][4189.487121393] 2025-12-04T10:02:38.5501691Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:02:38.5503235Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:02:38.549823] 2025-12-04T10:02:47.7384950Z 2025-12-04T10:02:47.7386246Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_e1870d365f721364_.log 2025-12-04T10:02:47.7388982Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_post_optim_event_overlap, test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2025-12-04T10:02:47.7390462Z 2025-12-04T10:02:47.7390969Z Finished distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2025-12-04 10:02:47.738077][4198.67571189], took 0.15min 2025-12-04T10:02:47.7510187Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-b60f2694fe7d2c52.xml 2025-12-04T10:02:47.8509939Z Running distributed/test_run 1/1 ... [2025-12-04 10:02:47.850636][4198.788272976] 2025-12-04T10:02:47.8510506Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:02:47.8511774Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_run.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:02:47.851003] 2025-12-04T10:02:51.6750315Z 2025-12-04T10:02:51.6751309Z distributed/test_run 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_run_1.1_048c18016597c35d_.log 2025-12-04T10:02:51.6753649Z Running 4 items in this shard: test/distributed/test_run.py::RunTest::test_config_from_args_signals_to_handle, test/distributed/test_run.py::RunTest::test_launch_agent_sets_environment_variable, test/distributed/test_run.py::RunTest::test_signals_to_handle_custom, test/distributed/test_run.py::RunTest::test_signals_to_handle_default 2025-12-04T10:02:51.6755364Z 2025-12-04T10:02:51.6755681Z Finished distributed/test_run 1/1 ... [2025-12-04 10:02:51.674694][4202.612324185], took 0.06min 2025-12-04T10:02:51.6877861Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_run/distributed.test_run-ff2c644d612e7850.xml 2025-12-04T10:02:51.7239906Z Running distributed/tensor/test_math_ops 1/1 ... [2025-12-04 10:02:51.723448][4202.661084891] 2025-12-04T10:02:51.7240530Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:02:51.7241771Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:02:51.723780] 2025-12-04T10:06:12.5418796Z 2025-12-04T10:06:12.5419974Z distributed/tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_math_ops_1.1_3bc3d9be883f9d0b_.log 2025-12-04T10:06:12.5447648Z Running 54 items in this shard: test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_conj_complex_dtensor, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_cumsum, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_foreach_add_different_mesh, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm_different_mesh, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm_partial, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_histc, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd_req_grad, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_linalg_eigh, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_logsumexp, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_matching_partial_reduction_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_partial_reduction_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_rotary_embedding_complex_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_std, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_topk, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_upsampling, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_vector_norm, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_vector_norm_partial, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_conj_complex_dtensor, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_cumsum, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_foreach_add_different_mesh, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_foreach_norm, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_foreach_norm_different_mesh, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_foreach_norm_partial, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_histc, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_layer_norm_bwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_layer_norm_bwd_req_grad, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_layer_norm_fwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_linalg_eigh, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_linear_op_reductions, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_logsumexp, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_matching_partial_reduction_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_mean, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_nll_loss_and_cross_entropy, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_partial_reduction_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_rotary_embedding_complex_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_shard0_svd, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_shard_math_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_softmax_fwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_softmax_with_bwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_std, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_topk, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_upsampling, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_vector_norm, test/distributed/tensor/test_math_ops.py::DistMathOpsTestWithLocalTensor::test_vector_norm_partial 2025-12-04T10:06:12.5470995Z 2025-12-04T10:06:12.5471398Z Finished distributed/tensor/test_math_ops 1/1 ... [2025-12-04 10:06:12.541458][4403.479093055], took 3.35min 2025-12-04T10:06:12.5550495Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_math_ops/distributed.tensor.test_math_ops-f7126468d0203b5f.xml 2025-12-04T10:06:12.8863591Z Uploading artifacts took 0.25 seconds 2025-12-04T10:06:12.8866890Z Running distributed/fsdp/test_utils 1/1 ... [2025-12-04 10:06:12.886491][4403.82412707] 2025-12-04T10:06:12.8867581Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:06:12.8870405Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:06:12.886833] 2025-12-04T10:06:17.9640467Z 2025-12-04T10:06:17.9641518Z distributed/fsdp/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_utils_1.1_849efa648c4a4b77_.log 2025-12-04T10:06:17.9644842Z Running 5 items in this shard: test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_cpu_cuda_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_device_list0_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_device_list1_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_packed_sequence_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_replace_by_prefix_cuda 2025-12-04T10:06:17.9647199Z 2025-12-04T10:06:17.9647578Z Finished distributed/fsdp/test_utils 1/1 ... [2025-12-04 10:06:17.963570][4408.901201437], took 0.08min 2025-12-04T10:06:17.9771609Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_utils/distributed.fsdp.test_utils-62adb141232f9dae.xml 2025-12-04T10:06:18.0102375Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2025-12-04 10:06:18.009615][4408.947252609] 2025-12-04T10:06:18.0103088Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:06:18.0104424Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:06:18.009951] 2025-12-04T10:08:09.8128720Z 2025-12-04T10:08:09.8130078Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_5915eea0acf0b33b_.log 2025-12-04T10:08:09.8143462Z Running 16 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_float64_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_0_attention_wv__layers_0_feed_forward_w1__layers_1_feed_forward_w2__layers_1_ffn_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_1_ffn_norm__norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying 2025-12-04T10:08:09.8156597Z 2025-12-04T10:08:09.8157074Z Finished distributed/tensor/parallel/test_tp_examples 1/1 ... [2025-12-04 10:08:09.812769][4520.750403496], took 1.86min 2025-12-04T10:08:09.8270223Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-9094c50849cf3609.xml 2025-12-04T10:08:09.9082261Z Running distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2025-12-04 10:08:09.907524][4520.845161127] 2025-12-04T10:08:09.9083241Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:08:09.9084603Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:08:09.907849] 2025-12-04T10:13:51.7271822Z 2025-12-04T10:13:51.7273156Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_4d95b4091bc3b359_.log 2025-12-04T10:13:51.7286509Z Running 18 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompileCompute::test_disable_compiling_hooks, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_compiled_autograd_ctx, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_recompiles_on_fsdp_layers, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_trace_use_training_state, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True_graph_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_trace_fsdp_copy_, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True_graph_partition 2025-12-04T10:13:51.7298649Z 2025-12-04T10:13:51.7299154Z Finished distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2025-12-04 10:13:51.727197][4862.664831892], took 5.70min 2025-12-04T10:13:51.7413590Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_compile/distributed._composable.fsdp.test_fully_shard_compile-0fa50fae7281ecc5.xml 2025-12-04T10:13:51.8215719Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2025-12-04 10:13:51.820908][4862.75854461] 2025-12-04T10:13:51.8216442Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:13:51.8217799Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:13:51.821246] 2025-12-04T10:14:24.4692433Z 2025-12-04T10:14:24.4697167Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_463e91c12def2aed_.log 2025-12-04T10:14:24.4703137Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2025-12-04T10:14:24.4707210Z 2025-12-04T10:14:24.4708407Z Finished distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2025-12-04 10:14:24.468721][4895.406351816], took 0.54min 2025-12-04T10:14:24.4835273Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-db4a02d440fa1c43.xml 2025-12-04T10:14:24.5726699Z Running distributed/checkpoint/test_pg_transport 1/1 ... [2025-12-04 10:14:24.572007][4895.509643943] 2025-12-04T10:14:24.5727939Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:14:24.5729532Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_pg_transport.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:14:24.572340] 2025-12-04T10:14:38.7731865Z 2025-12-04T10:14:38.7732998Z distributed/checkpoint/test_pg_transport 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_pg_transport_1.1_26ddb76dc3b5e83f_.log 2025-12-04T10:14:38.7757811Z Running 21 items in this shard: test/distributed/checkpoint/test_pg_transport.py::PgTransportCPU::test_pg_transport, test/distributed/checkpoint/test_pg_transport.py::PgTransportCPU::test_pg_transport_with_mixed_content, test/distributed/checkpoint/test_pg_transport.py::PgTransportCPU::test_pg_transport_with_sharded_tensor, test/distributed/checkpoint/test_pg_transport.py::PgTransportGPU::test_pg_transport, test/distributed/checkpoint/test_pg_transport.py::PgTransportGPU::test_pg_transport_with_mixed_content, test/distributed/checkpoint/test_pg_transport.py::PgTransportGPU::test_pg_transport_with_sharded_tensor, test/distributed/checkpoint/test_pg_transport.py::TestCastTensor::test_cast_tensor_different_dtypes, test/distributed/checkpoint/test_pg_transport.py::TestCastTensor::test_cast_tensor_with_offset, test/distributed/checkpoint/test_pg_transport.py::TestCastTensor::test_cast_tensor_with_stride, test/distributed/checkpoint/test_pg_transport.py::TestPrepareTensor::test_prepare_tensor_basic, test/distributed/checkpoint/test_pg_transport.py::TestPrepareTensor::test_prepare_tensor_different_shapes, test/distributed/checkpoint/test_pg_transport.py::TestPrepareTensor::test_prepare_tensor_with_stride, test/distributed/checkpoint/test_pg_transport.py::TestPrepareStateDict::test_prepare_state_dict_basic, test/distributed/checkpoint/test_pg_transport.py::TestPrepareStateDict::test_prepare_state_dict_nested, test/distributed/checkpoint/test_pg_transport.py::TestPrepareStateDict::test_prepare_state_dict_with_non_tensor_values, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_recv_checkpoint_basic, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_recv_checkpoint_with_state_dict_callback, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_send_checkpoint_basic, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_send_checkpoint_empty_state_dict, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_send_checkpoint_with_non_tensor_values, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportEdgeCases::test_send_checkpoint_with_cpu_tensors 2025-12-04T10:14:38.7780051Z 2025-12-04T10:14:38.7780848Z Finished distributed/checkpoint/test_pg_transport 1/1 ... [2025-12-04 10:14:38.773125][4909.710757945], took 0.24min 2025-12-04T10:14:38.7894875Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_pg_transport/distributed.checkpoint.test_pg_transport-f8a475bbd4225371.xml 2025-12-04T10:14:38.9035247Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2025-12-04 10:14:38.903131][4909.840768532] 2025-12-04T10:14:38.9036016Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:14:38.9037398Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:14:38.903470] 2025-12-04T10:15:10.8489625Z 2025-12-04T10:15:10.8491107Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_9054fdac15ee60e1_.log 2025-12-04T10:15:10.8499513Z Running 9 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_grad_acc_with_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_clamp_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_dataclass_input, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_float16_on_one_submodule, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_bf16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_fp16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_submodules_with_external_inputs 2025-12-04T10:15:10.8505868Z 2025-12-04T10:15:10.8506380Z Finished distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2025-12-04 10:15:10.848318][4941.785954452], took 0.53min 2025-12-04T10:15:10.8626045Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-5f953c8bff259762.xml 2025-12-04T10:15:10.9568099Z Running distributed/checkpoint/test_utils 1/1 ... [2025-12-04 10:15:10.956283][4941.893919978] 2025-12-04T10:15:10.9568729Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:15:10.9569999Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:15:10.956621] 2025-12-04T10:15:45.0600714Z 2025-12-04T10:15:45.0601802Z distributed/checkpoint/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_utils_1.1_345919bf14d8d525_.log 2025-12-04T10:15:45.0609603Z Running 16 items in this shard: test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_dcp_logger, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_flat_data, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_equals, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_hash, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_init_convert_offset, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_sharded_tensor_lookup, test/distributed/checkpoint/test_utils.py::TestReaderView::testAllRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testLongRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testLongReadinto, test/distributed/checkpoint/test_utils.py::TestReaderView::testShortRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testShortReadinto, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_barrier, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_broadcast_object_global_local_mismatch, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_broadcast_object_with_nonzero_coordinator, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_gather_object, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_scatter_object 2025-12-04T10:15:45.0616864Z 2025-12-04T10:15:45.0617247Z Finished distributed/checkpoint/test_utils 1/1 ... [2025-12-04 10:15:45.059558][4975.997193224], took 0.57min 2025-12-04T10:15:45.0736970Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_utils/distributed.checkpoint.test_utils-3867a40d83a2b51a.xml 2025-12-04T10:15:45.1669236Z Running distributed/checkpoint/_experimental/test_checkpoint_process 1/1 ... [2025-12-04 10:15:45.166321][4976.103958697] 2025-12-04T10:15:45.1670006Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:15:45.1671461Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_checkpoint_process.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:15:45.166655] 2025-12-04T10:16:18.6598050Z 2025-12-04T10:16:18.6599480Z distributed/checkpoint/_experimental/test_checkpoint_process 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_checkpoint_process_1.1_35f3f306f7f6005f_.log 2025-12-04T10:16:18.6610557Z Running 15 items in this shard: test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestRequestTypes::test_request_type_enum, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestRequestTypes::test_worker_request, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestRequestTypes::test_worker_response, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcessConfig::test_custom_options, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcessConfig::test_default_options, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_checkpoint_process_initialization, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_checkpoint_write_future_state_dict, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_checkpoint_write_sync_state_dict, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_checkpoint_write_with_kwargs, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_communication_error_handling, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_forced_termination, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_graceful_termination, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_shared_memory_tensor_ipc, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_subprocess_initialization_failure, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_subprocess_initialization_timeout 2025-12-04T10:16:18.6620271Z 2025-12-04T10:16:18.6620799Z Finished distributed/checkpoint/_experimental/test_checkpoint_process 1/1 ... [2025-12-04 10:16:18.659345][5009.596980843], took 0.56min 2025-12-04T10:16:18.6743967Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint._experimental.test_checkpoint_process/distributed.checkpoint._experimental.test_checkpoint_process-8b0b149c7b33e94c.xml 2025-12-04T10:16:18.7606656Z Running distributed/tensor/test_view_ops 1/1 ... [2025-12-04 10:16:18.760078][5009.697715243] 2025-12-04T10:16:18.7607269Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:16:18.7608518Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:16:18.760422] 2025-12-04T10:19:13.7781534Z 2025-12-04T10:19:13.7782593Z distributed/tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_view_ops_1.1_75425393e05735f8_.log 2025-12-04T10:19:13.7792933Z Running 20 items in this shard: test/distributed/tensor/test_view_ops.py::TestViewOps::test_complex_view_ops, test/distributed/tensor/test_view_ops.py::TestViewOps::test_dtensor_view_op_uneven, test/distributed/tensor/test_view_ops.py::TestViewOps::test_illegal_views, test/distributed/tensor/test_view_ops.py::TestViewOps::test_squeeze_, test/distributed/tensor/test_view_ops.py::TestViewOps::test_storage_offset_shard_dim0_slice_dim1, test/distributed/tensor/test_view_ops.py::TestViewOps::test_storage_offset_shard_dim1_slice_dim0, test/distributed/tensor/test_view_ops.py::TestViewOps::test_storage_offset_slice, test/distributed/tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/tensor/test_view_ops.py::TestViewOps::test_view_ops, test/distributed/tensor/test_view_ops.py::TestViewOps::test_view_redistribution, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_complex_view_ops, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_dtensor_view_op_uneven, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_illegal_views, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_squeeze_, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_storage_offset_shard_dim0_slice_dim1, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_storage_offset_shard_dim1_slice_dim0, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_storage_offset_slice, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_view_groups, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_view_ops, test/distributed/tensor/test_view_ops.py::TestViewOpsWithLocalTensor::test_view_redistribution 2025-12-04T10:19:13.7801858Z 2025-12-04T10:19:13.7802228Z Finished distributed/tensor/test_view_ops 1/1 ... [2025-12-04 10:19:13.777842][5184.71547708], took 2.92min 2025-12-04T10:19:13.7923765Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.tensor.test_view_ops/distributed.tensor.test_view_ops-04bcb1d8735e8634.xml 2025-12-04T10:19:13.8790178Z Running distributed/fsdp/test_fsdp_state_dict 2/2 ... [2025-12-04 10:19:13.878495][5184.816131913] 2025-12-04T10:19:13.8790814Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:19:13.8792125Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:19:13.878852] 2025-12-04T10:27:45.0508742Z 2025-12-04T10:27:45.0510281Z distributed/fsdp/test_fsdp_state_dict 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_1eec504a312e52ef_.log 2025-12-04T10:27:45.0605048Z Running 101 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_sharded_load_multi_backend_pg, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config 2025-12-04T10:27:45.0694542Z 2025-12-04T10:27:45.0694918Z Finished distributed/fsdp/test_fsdp_state_dict 2/2 ... [2025-12-04 10:27:45.052304][5695.989936919], took 8.52min 2025-12-04T10:27:45.0696237Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-0e669e7d48e3457f.xml 2025-12-04T10:27:45.4125185Z Uploading artifacts took 0.23 seconds 2025-12-04T10:27:45.4126609Z Running distributed/test_distributed_spawn 2/9 ... [2025-12-04 10:27:45.412517][5696.35015384] 2025-12-04T10:27:45.4128999Z Running distributed tests for the test backend with env init_method 2025-12-04T10:27:45.4130733Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:27:45.4134536Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:27:45.413272] 2025-12-04T10:27:48.9983116Z 2025-12-04T10:27:48.9984246Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_2b1bd9557bd2920d_.log 2025-12-04T10:27:48.9985323Z Running 0 items in this shard: 2025-12-04T10:27:48.9985559Z 2025-12-04T10:27:48.9986208Z Running distributed tests for the test backend with file init_method 2025-12-04T10:27:48.9988188Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:27:48.9992258Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:27:48.999020] 2025-12-04T10:27:52.5816431Z 2025-12-04T10:27:52.5817759Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_ee8af3844830e146_.log 2025-12-04T10:27:52.5818911Z Running 0 items in this shard: 2025-12-04T10:27:52.5819143Z 2025-12-04T10:27:52.5819398Z Running distributed tests for the mpi backend with env init_method 2025-12-04T10:27:52.7104645Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:27:52.7106421Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:27:52.710238] 2025-12-04T10:27:56.8950699Z 2025-12-04T10:27:56.8951824Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_f1f2ba629914f596_.log 2025-12-04T10:27:56.8952922Z Running 0 items in this shard: 2025-12-04T10:27:56.8953255Z Running 0 items in this shard: 2025-12-04T10:27:56.8953592Z Running 0 items in this shard: 2025-12-04T10:27:56.8953802Z 2025-12-04T10:27:56.8957665Z Running distributed tests for the mpi backend with file init_method 2025-12-04T10:27:57.0220838Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:27:57.0222385Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:27:57.021818] 2025-12-04T10:28:01.2253375Z 2025-12-04T10:28:01.2254534Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_f4e4b80156692b3f_.log 2025-12-04T10:28:01.2255580Z Running 0 items in this shard: 2025-12-04T10:28:01.2256230Z Running 0 items in this shard: 2025-12-04T10:28:01.2256553Z Running 0 items in this shard: 2025-12-04T10:28:01.2256759Z 2025-12-04T10:28:01.2258699Z Running distributed tests for the nccl backend with env init_method 2025-12-04T10:28:01.2260190Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:28:01.2263657Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:28:01.226162] 2025-12-04T10:31:43.8285087Z 2025-12-04T10:31:43.8286460Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_c85ca92b15a87ea6_.log 2025-12-04T10:31:43.8302967Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2025-12-04T10:31:43.8319254Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2025-12-04T10:31:43.8320739Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2025-12-04T10:31:43.8321963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2025-12-04T10:31:43.8323205Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2025-12-04T10:31:43.8324854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-12-04T10:31:43.8326125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-12-04T10:31:43.8327280Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2025-12-04T10:31:43.8328525Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-12-04T10:31:43.8329881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2025-12-04T10:31:43.8331124Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2025-12-04T10:31:43.8332228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2025-12-04T10:31:43.8333428Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2025-12-04T10:31:43.8334760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2025-12-04T10:31:43.8336177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-12-04T10:31:43.8337334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2025-12-04T10:31:43.8338493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2025-12-04T10:31:43.8339696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2025-12-04T10:31:43.8342279Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2025-12-04T10:31:43.8343499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2025-12-04T10:31:43.8344653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2025-12-04T10:31:43.8345814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2025-12-04T10:31:43.8347149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2025-12-04T10:31:43.8348745Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2025-12-04T10:31:43.8350160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2025-12-04T10:31:43.8351674Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2025-12-04T10:31:43.8352995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2025-12-04T10:31:43.8354332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2025-12-04T10:31:43.8355508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2025-12-04T10:31:43.8356170Z 2025-12-04T10:31:43.8356423Z Running distributed tests for the nccl backend with file init_method 2025-12-04T10:31:43.8356943Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:31:43.8358330Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:31:43.830156] 2025-12-04T10:35:26.5440274Z 2025-12-04T10:35:26.5441344Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_7c6c5eed862dee3b_.log 2025-12-04T10:35:26.5457204Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2025-12-04T10:35:26.5473047Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2025-12-04T10:35:26.5474474Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2025-12-04T10:35:26.5475757Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2025-12-04T10:35:26.5477046Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2025-12-04T10:35:26.5478375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-12-04T10:35:26.5479730Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-12-04T10:35:26.5480863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2025-12-04T10:35:26.5482070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-12-04T10:35:26.5483393Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2025-12-04T10:35:26.5484589Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2025-12-04T10:35:26.5485682Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2025-12-04T10:35:26.5486838Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2025-12-04T10:35:26.5488117Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2025-12-04T10:35:26.5489387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-12-04T10:35:26.5490549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2025-12-04T10:35:26.5491707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2025-12-04T10:35:26.5492984Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2025-12-04T10:35:26.5494133Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2025-12-04T10:35:26.5495306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2025-12-04T10:35:26.5496465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2025-12-04T10:35:26.5497627Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2025-12-04T10:35:26.5498964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2025-12-04T10:35:26.5500272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2025-12-04T10:35:26.5501669Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2025-12-04T10:35:26.5503067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2025-12-04T10:35:26.5504351Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2025-12-04T10:35:26.5505594Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2025-12-04T10:35:26.5506749Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2025-12-04T10:35:26.5507463Z 2025-12-04T10:35:26.5507881Z Running distributed tests for the gloo backend with env init_method 2025-12-04T10:35:26.5508401Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:35:26.5509759Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:35:26.545583] 2025-12-04T10:39:19.7834660Z 2025-12-04T10:39:19.7835802Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_f3ae3ad69e49db65_.log 2025-12-04T10:39:19.7852724Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2025-12-04T10:39:19.7868696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2025-12-04T10:39:19.7870022Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2025-12-04T10:39:19.7871300Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2025-12-04T10:39:19.7872602Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2025-12-04T10:39:19.7873926Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-12-04T10:39:19.7875185Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-12-04T10:39:19.7876340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2025-12-04T10:39:19.7877583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-12-04T10:39:19.7878935Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2025-12-04T10:39:19.7880274Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2025-12-04T10:39:19.7881346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2025-12-04T10:39:19.7882499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2025-12-04T10:39:19.7883784Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2025-12-04T10:39:19.7885068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-12-04T10:39:19.7886234Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2025-12-04T10:39:19.7887478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2025-12-04T10:39:19.7888693Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2025-12-04T10:39:19.7889852Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2025-12-04T10:39:19.7891009Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2025-12-04T10:39:19.7892170Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2025-12-04T10:39:19.7893333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2025-12-04T10:39:19.7894673Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2025-12-04T10:39:19.7896017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2025-12-04T10:39:19.7897369Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2025-12-04T10:39:19.7898769Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2025-12-04T10:39:19.7900091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2025-12-04T10:39:19.7901318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2025-12-04T10:39:19.7902470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2025-12-04T10:39:19.7903119Z 2025-12-04T10:39:19.7903366Z Running distributed tests for the gloo backend with file init_method 2025-12-04T10:39:19.7903872Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:39:19.7905191Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:39:19.785251] 2025-12-04T10:43:12.6779974Z 2025-12-04T10:43:12.6781115Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_2ba023b93e4a551d_.log 2025-12-04T10:43:12.6798695Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2025-12-04T10:43:12.6814484Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2025-12-04T10:43:12.6815785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2025-12-04T10:43:12.6817000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2025-12-04T10:43:12.6818254Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2025-12-04T10:43:12.6819541Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-12-04T10:43:12.6820766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-12-04T10:43:12.6821886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2025-12-04T10:43:12.6823099Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-12-04T10:43:12.6824817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2025-12-04T10:43:12.6826078Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2025-12-04T10:43:12.6827185Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2025-12-04T10:43:12.6828485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2025-12-04T10:43:12.6829818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2025-12-04T10:43:12.6831136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-12-04T10:43:12.6832442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2025-12-04T10:43:12.6833643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2025-12-04T10:43:12.6834888Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2025-12-04T10:43:12.6836078Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2025-12-04T10:43:12.6837268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2025-12-04T10:43:12.6838455Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2025-12-04T10:43:12.6839764Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2025-12-04T10:43:12.6841142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2025-12-04T10:43:12.6842460Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2025-12-04T10:43:12.6843807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2025-12-04T10:43:12.6845268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2025-12-04T10:43:12.6846553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2025-12-04T10:43:12.6847783Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2025-12-04T10:43:12.6848925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2025-12-04T10:43:12.6849566Z 2025-12-04T10:43:12.6849958Z Finished distributed/test_distributed_spawn 2/9 ... [2025-12-04 10:43:12.678768][6623.616402389], took 15.45min 2025-12-04T10:43:12.6949076Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-975ac1cd2ed3ea0d.xml 2025-12-04T10:43:12.7769542Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8babbac9d40e37c7.xml 2025-12-04T10:43:12.8067375Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4f717ad1d99b753.xml 2025-12-04T10:43:12.8350363Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-337a55931fe74487.xml 2025-12-04T10:43:12.8616654Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90b3763c34aa2960.xml 2025-12-04T10:43:12.8864933Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d0f38b19d35d9f0.xml 2025-12-04T10:43:12.9125649Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a57a7c1f6c446d8.xml 2025-12-04T10:43:12.9433837Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9a464945bf1177b.xml 2025-12-04T10:43:12.9767685Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1497afbb3545da85.xml 2025-12-04T10:43:13.0068200Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cbb5e6acdeef21a7.xml 2025-12-04T10:43:13.0367386Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e37edf773eeec4cb.xml 2025-12-04T10:43:13.0663769Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9cafccc700b8925.xml 2025-12-04T10:43:13.0952115Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b34ce6e907f1d03.xml 2025-12-04T10:43:13.1414029Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1450c3d527670c7.xml 2025-12-04T10:43:13.1764120Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85297e8464ee07d1.xml 2025-12-04T10:43:13.2472260Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aff79cf3432fbb8f.xml 2025-12-04T10:43:13.2786305Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8141293960cc76b4.xml 2025-12-04T10:43:13.3110921Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ed1d3854cb079b5.xml 2025-12-04T10:43:13.3447592Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f5ee7b2c181b77f.xml 2025-12-04T10:43:13.3713887Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3de0668036a4d46a.xml 2025-12-04T10:43:13.4054206Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ed12ff4e601d4a5.xml 2025-12-04T10:43:13.4351045Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55742e93980fa69f.xml 2025-12-04T10:43:13.4646462Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d1649152d68be9c.xml 2025-12-04T10:43:13.4933911Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3c70db6f7d0cc8f.xml 2025-12-04T10:43:13.5253744Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0fb83c60b047974.xml 2025-12-04T10:43:13.5565124Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78a8a7c12b3fdbcd.xml 2025-12-04T10:43:13.5903460Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88531b1bf42e91d7.xml 2025-12-04T10:43:13.6204495Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22e23e45bc80123f.xml 2025-12-04T10:43:13.6526425Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f92f6113f8978240.xml 2025-12-04T10:43:13.6838863Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15dc7f75169494c0.xml 2025-12-04T10:43:13.7166526Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e73ef2b0e35926a2.xml 2025-12-04T10:43:13.7463910Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f598c7195f78ef8.xml 2025-12-04T10:43:13.7816583Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88b5e238fde7cdb9.xml 2025-12-04T10:43:13.8145404Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8880a41f88a6bb0.xml 2025-12-04T10:43:13.8604079Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-365036945f26d41b.xml 2025-12-04T10:43:13.8894436Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e819ba4cf6e31f8.xml 2025-12-04T10:43:13.9226295Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40bc39b1652660d2.xml 2025-12-04T10:43:13.9585947Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f9afae11c179070.xml 2025-12-04T10:43:13.9909946Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3c746330754bbc2.xml 2025-12-04T10:43:14.0216435Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b68ead304b495067.xml 2025-12-04T10:43:14.0517781Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7886ccd5b9eeb78.xml 2025-12-04T10:43:14.0826954Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21598fac9ec33b36.xml 2025-12-04T10:43:14.1126359Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45fc07c2582c64de.xml 2025-12-04T10:43:14.1409791Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9a64b3d5828bdd7.xml 2025-12-04T10:43:14.1750939Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f441409f584e7860.xml 2025-12-04T10:43:14.2086820Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02d51cd92aaffd99.xml 2025-12-04T10:43:14.2387309Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd7c20c72a58c490.xml 2025-12-04T10:43:14.2713257Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a885f907c0ea15f.xml 2025-12-04T10:43:14.3048306Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd6772c504021b1c.xml 2025-12-04T10:43:14.3415754Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b129e835aba47f3.xml 2025-12-04T10:43:14.3722776Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dca90ceb118f7f51.xml 2025-12-04T10:43:14.4016049Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4c7cdb746d63750.xml 2025-12-04T10:43:14.4418766Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b62455b26fd75a3.xml 2025-12-04T10:43:14.4710885Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c6c3445d3489ed7.xml 2025-12-04T10:43:14.5018336Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc9043c9b1f5531d.xml 2025-12-04T10:43:14.5331044Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fe0c34ac3a70b57.xml 2025-12-04T10:43:14.5654272Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93b994665f730626.xml 2025-12-04T10:43:14.5978741Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e40c62350b42648a.xml 2025-12-04T10:43:14.6298310Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c79494752e7d5ea.xml 2025-12-04T10:43:14.6603988Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6369d472a2cdae13.xml 2025-12-04T10:43:14.6967236Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-342afb36b7c5d09b.xml 2025-12-04T10:43:14.7311103Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f18aaac9bd89cd2c.xml 2025-12-04T10:43:14.7602891Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51c0d6585102aef7.xml 2025-12-04T10:43:14.8146270Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e2c29cc0f67c6e0.xml 2025-12-04T10:43:14.8470836Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e04c95a8c2fac8bc.xml 2025-12-04T10:43:14.9005498Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2143f4a3910ea5d.xml 2025-12-04T10:43:14.9345889Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a623b45ea2222a4a.xml 2025-12-04T10:43:14.9696523Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b489fc84ac56c90a.xml 2025-12-04T10:43:15.0044702Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c606f93062a316ee.xml 2025-12-04T10:43:15.0353997Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13409fa898261f4d.xml 2025-12-04T10:43:15.0711020Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-822c6e3f32fc24f4.xml 2025-12-04T10:43:15.1046569Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee05bfd466991d42.xml 2025-12-04T10:43:15.1382775Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-997ca84c08e13a4f.xml 2025-12-04T10:43:15.1697829Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1650b0a1cf5101b.xml 2025-12-04T10:43:15.2067660Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f17cf2fe88278b7.xml 2025-12-04T10:43:15.2406785Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d173d160413f56ff.xml 2025-12-04T10:43:15.2689331Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c034757c66530030.xml 2025-12-04T10:43:15.3004420Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19e760457ef6ff92.xml 2025-12-04T10:43:15.3374249Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78ba10be7d074c23.xml 2025-12-04T10:43:15.3734947Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b4bba108f9cb06d.xml 2025-12-04T10:43:15.4063738Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-049ae9466a303b5c.xml 2025-12-04T10:43:15.4377853Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44b7dc32fb33d263.xml 2025-12-04T10:43:15.4672376Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75e1e407437545e3.xml 2025-12-04T10:43:15.5184848Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70326d0de199e967.xml 2025-12-04T10:43:15.5503949Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-537e26a62ece27c3.xml 2025-12-04T10:43:15.5815495Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-276b8324a6920515.xml 2025-12-04T10:43:15.6151108Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2263791b16c51a6.xml 2025-12-04T10:43:15.6473338Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebaf98e72782bada.xml 2025-12-04T10:43:15.6796503Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51f4be7c600c130e.xml 2025-12-04T10:43:15.7157410Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64a85d7811c12838.xml 2025-12-04T10:43:15.7470543Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24ebf91103f6fea1.xml 2025-12-04T10:43:15.7766396Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-206af608512f699e.xml 2025-12-04T10:43:15.8070975Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f82acc534672cdf.xml 2025-12-04T10:43:15.8374776Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bc0e514e7eb9947.xml 2025-12-04T10:43:15.8903587Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5301f1689efddffd.xml 2025-12-04T10:43:15.9232336Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39f8f74b2c4bb356.xml 2025-12-04T10:43:15.9546631Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4deb287bf0f15502.xml 2025-12-04T10:43:15.9873635Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-69b712ded204926d.xml 2025-12-04T10:43:16.0187009Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac8d4732ea94303d.xml 2025-12-04T10:43:16.0486829Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b15b1dbca96c31dd.xml 2025-12-04T10:43:16.0815647Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5235361330778453.xml 2025-12-04T10:43:16.1678364Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d366d5954251fcda.xml 2025-12-04T10:43:16.2008293Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b47d5bdbdea81123.xml 2025-12-04T10:43:16.2449841Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a24f6ee3b330fe2.xml 2025-12-04T10:43:16.2764379Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6ba4b460a331f8f.xml 2025-12-04T10:43:16.3050664Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-676fe7068d5b8cce.xml 2025-12-04T10:43:16.3325149Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a360a85ac03dda59.xml 2025-12-04T10:43:16.3632727Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1708cca9dc0e798.xml 2025-12-04T10:43:16.3944857Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a8ced54e15b6002.xml 2025-12-04T10:43:16.4265558Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c9833d98fcaa164.xml 2025-12-04T10:43:16.4593475Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eec63fa6ac957019.xml 2025-12-04T10:43:16.4933186Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9032fc63d800720d.xml 2025-12-04T10:43:16.5247617Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9237cfd956a6ea6.xml 2025-12-04T10:43:16.5597874Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d3dac1ff57d770a.xml 2025-12-04T10:43:16.5953933Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e71aa694c478fc1e.xml 2025-12-04T10:43:16.6273792Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3303d981481b9d1f.xml 2025-12-04T10:43:16.6631461Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10e39578cdaa15c7.xml 2025-12-04T10:43:16.6925886Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1b45f0468676a59.xml 2025-12-04T10:43:16.7248857Z Running distributed/test_distributed_spawn 5/9 ... [2025-12-04 10:43:16.724231][6627.661867718] 2025-12-04T10:43:16.7249625Z Running distributed tests for the test backend with env init_method 2025-12-04T10:43:16.7250128Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:43:16.7251735Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:43:16.724979] 2025-12-04T10:43:20.3116156Z 2025-12-04T10:43:20.3117279Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_ad3d12242976253d_.log 2025-12-04T10:43:20.3118399Z Running 0 items in this shard: 2025-12-04T10:43:20.3118632Z 2025-12-04T10:43:20.3125361Z Running distributed tests for the test backend with file init_method 2025-12-04T10:43:20.3126913Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:43:20.3130923Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:43:20.312899] 2025-12-04T10:43:23.8988452Z 2025-12-04T10:43:23.8989891Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_b3816ec15751d402_.log 2025-12-04T10:43:23.8990989Z Running 0 items in this shard: 2025-12-04T10:43:23.8991230Z 2025-12-04T10:43:23.8992142Z Running distributed tests for the mpi backend with env init_method 2025-12-04T10:43:24.0304746Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:43:24.0308049Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:43:24.030390] 2025-12-04T10:43:28.2162786Z 2025-12-04T10:43:28.2164099Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_e692ef08a9a81937_.log 2025-12-04T10:43:28.2165155Z Running 0 items in this shard: 2025-12-04T10:43:28.2165491Z Running 0 items in this shard: 2025-12-04T10:43:28.2165801Z Running 0 items in this shard: 2025-12-04T10:43:28.2166015Z 2025-12-04T10:43:28.2166380Z Running distributed tests for the mpi backend with file init_method 2025-12-04T10:43:28.3452058Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:43:28.3454810Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:43:28.345128] 2025-12-04T10:43:32.5209181Z 2025-12-04T10:43:32.5210344Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_466c47db11e78e35_.log 2025-12-04T10:43:32.5211427Z Running 0 items in this shard: 2025-12-04T10:43:32.5211761Z Running 0 items in this shard: 2025-12-04T10:43:32.5212075Z Running 0 items in this shard: 2025-12-04T10:43:32.5212287Z 2025-12-04T10:43:32.5213835Z Running distributed tests for the nccl backend with env init_method 2025-12-04T10:43:32.5215425Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:43:32.5219460Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:43:32.521750] 2025-12-04T10:48:16.0155450Z 2025-12-04T10:48:16.0156933Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_10357f3c9e30db03_.log 2025-12-04T10:48:16.0176858Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-12-04T10:48:16.0197016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2025-12-04T10:48:16.0198476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2025-12-04T10:48:16.0199967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2025-12-04T10:48:16.0201214Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2025-12-04T10:48:16.0202397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2025-12-04T10:48:16.0203550Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2025-12-04T10:48:16.0204686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2025-12-04T10:48:16.0205842Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2025-12-04T10:48:16.0207054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2025-12-04T10:48:16.0208198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2025-12-04T10:48:16.0209388Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2025-12-04T10:48:16.0210701Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2025-12-04T10:48:16.0211950Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2025-12-04T10:48:16.0213121Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2025-12-04T10:48:16.0214293Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2025-12-04T10:48:16.0215488Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2025-12-04T10:48:16.0216714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2025-12-04T10:48:16.0217844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2025-12-04T10:48:16.0219121Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2025-12-04T10:48:16.0220459Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2025-12-04T10:48:16.0221777Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2025-12-04T10:48:16.0223061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-12-04T10:48:16.0224746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-12-04T10:48:16.0225961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2025-12-04T10:48:16.0227133Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2025-12-04T10:48:16.0228388Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2025-12-04T10:48:16.0229667Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2025-12-04T10:48:16.0230815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2025-12-04T10:48:16.0232105Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2025-12-04T10:48:16.0233303Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2025-12-04T10:48:16.0234591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2025-12-04T10:48:16.0235941Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2025-12-04T10:48:16.0237063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2025-12-04T10:48:16.0238249Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2025-12-04T10:48:16.0239614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-12-04T10:48:16.0240785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2025-12-04T10:48:16.0242040Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-12-04T10:48:16.0242719Z 2025-12-04T10:48:16.0242960Z Running distributed tests for the nccl backend with file init_method 2025-12-04T10:48:16.0243450Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:48:16.0244739Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:48:16.017393] 2025-12-04T10:52:59.5996251Z 2025-12-04T10:52:59.5997332Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_2cc71357f7a921e0_.log 2025-12-04T10:52:59.6017511Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-12-04T10:52:59.6038042Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2025-12-04T10:52:59.6039530Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2025-12-04T10:52:59.6040924Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2025-12-04T10:52:59.6042160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2025-12-04T10:52:59.6043334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2025-12-04T10:52:59.6044502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2025-12-04T10:52:59.6045652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2025-12-04T10:52:59.6046794Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2025-12-04T10:52:59.6047973Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2025-12-04T10:52:59.6049134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2025-12-04T10:52:59.6050335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2025-12-04T10:52:59.6051763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2025-12-04T10:52:59.6053007Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2025-12-04T10:52:59.6054186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2025-12-04T10:52:59.6055347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2025-12-04T10:52:59.6056565Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2025-12-04T10:52:59.6057810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2025-12-04T10:52:59.6058944Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2025-12-04T10:52:59.6060253Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2025-12-04T10:52:59.6061593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2025-12-04T10:52:59.6062900Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2025-12-04T10:52:59.6064219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-12-04T10:52:59.6065367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-12-04T10:52:59.6066520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2025-12-04T10:52:59.6067906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2025-12-04T10:52:59.6069119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2025-12-04T10:52:59.6070263Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2025-12-04T10:52:59.6071401Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2025-12-04T10:52:59.6072696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2025-12-04T10:52:59.6073898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2025-12-04T10:52:59.6075186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2025-12-04T10:52:59.6076522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2025-12-04T10:52:59.6077652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2025-12-04T10:52:59.6078788Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2025-12-04T10:52:59.6080153Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-12-04T10:52:59.6081306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2025-12-04T10:52:59.6082524Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-12-04T10:52:59.6083193Z 2025-12-04T10:52:59.6083486Z Running distributed tests for the gloo backend with env init_method 2025-12-04T10:52:59.6083971Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:52:59.6085235Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:52:59.601484] 2025-12-04T10:57:57.9825904Z 2025-12-04T10:57:57.9827087Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_403ffc079b9fc114_.log 2025-12-04T10:57:57.9849487Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-12-04T10:57:57.9870781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2025-12-04T10:57:57.9872160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2025-12-04T10:57:57.9873568Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2025-12-04T10:57:57.9874908Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2025-12-04T10:57:57.9876120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2025-12-04T10:57:57.9877391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2025-12-04T10:57:57.9878604Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2025-12-04T10:57:57.9879884Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2025-12-04T10:57:57.9881064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2025-12-04T10:57:57.9882210Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2025-12-04T10:57:57.9883407Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2025-12-04T10:57:57.9884687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2025-12-04T10:57:57.9885929Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2025-12-04T10:57:57.9887108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2025-12-04T10:57:57.9888272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2025-12-04T10:57:57.9889467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2025-12-04T10:57:57.9890775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2025-12-04T10:57:57.9891902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2025-12-04T10:57:57.9893163Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2025-12-04T10:57:57.9894500Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2025-12-04T10:57:57.9895805Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2025-12-04T10:57:57.9897087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-12-04T10:57:57.9899704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-12-04T10:57:57.9900931Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2025-12-04T10:57:57.9902045Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2025-12-04T10:57:57.9903158Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2025-12-04T10:57:57.9904264Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2025-12-04T10:57:57.9905383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2025-12-04T10:57:57.9906620Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2025-12-04T10:57:57.9908064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2025-12-04T10:57:57.9909339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2025-12-04T10:57:57.9910690Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2025-12-04T10:57:57.9911849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2025-12-04T10:57:57.9912984Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2025-12-04T10:57:57.9914157Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-12-04T10:57:57.9915400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2025-12-04T10:57:57.9916776Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-12-04T10:57:57.9917480Z 2025-12-04T10:57:57.9917735Z Running distributed tests for the gloo backend with file init_method 2025-12-04T10:57:57.9918252Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T10:57:57.9919610Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 10:57:57.984431] 2025-12-04T11:02:56.3521541Z 2025-12-04T11:02:56.3522745Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_6ee2fdeafd5e6202_.log 2025-12-04T11:02:56.3544235Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-12-04T11:02:56.3564602Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2025-12-04T11:02:56.3565952Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2025-12-04T11:02:56.3567342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2025-12-04T11:02:56.3568575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2025-12-04T11:02:56.3569753Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2025-12-04T11:02:56.3570893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2025-12-04T11:02:56.3572037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2025-12-04T11:02:56.3573267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2025-12-04T11:02:56.3574452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2025-12-04T11:02:56.3575600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2025-12-04T11:02:56.3576797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2025-12-04T11:02:56.3578078Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2025-12-04T11:02:56.3579315Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2025-12-04T11:02:56.3580489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2025-12-04T11:02:56.3581694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2025-12-04T11:02:56.3582887Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2025-12-04T11:02:56.3584106Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2025-12-04T11:02:56.3585270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2025-12-04T11:02:56.3586535Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2025-12-04T11:02:56.3588125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2025-12-04T11:02:56.3589605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2025-12-04T11:02:56.3590930Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-12-04T11:02:56.3592126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-12-04T11:02:56.3593330Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2025-12-04T11:02:56.3594487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2025-12-04T11:02:56.3595631Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2025-12-04T11:02:56.3596783Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2025-12-04T11:02:56.3597934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2025-12-04T11:02:56.3599313Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2025-12-04T11:02:56.3600568Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2025-12-04T11:02:56.3601775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2025-12-04T11:02:56.3603051Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2025-12-04T11:02:56.3604105Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2025-12-04T11:02:56.3606667Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2025-12-04T11:02:56.3607793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-12-04T11:02:56.3608964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2025-12-04T11:02:56.3610178Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2025-12-04T11:02:56.3610850Z 2025-12-04T11:02:56.3611237Z Finished distributed/test_distributed_spawn 5/9 ... [2025-12-04 11:02:56.353459][7807.291093882], took 19.66min 2025-12-04T11:02:56.3704942Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85365fcd3056dbf4.xml 2025-12-04T11:02:56.4484415Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af2626afcb0e4552.xml 2025-12-04T11:02:56.4710811Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a30a6d9cd45f8b4.xml 2025-12-04T11:02:56.4960935Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62bde0ef8091e7c6.xml 2025-12-04T11:02:56.5210630Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-101e5756265aa78e.xml 2025-12-04T11:02:56.5461731Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3553a63d7be9bc12.xml 2025-12-04T11:02:56.5751907Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b0298f7abe568d4.xml 2025-12-04T11:02:56.6010129Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e4e163fd1daa635.xml 2025-12-04T11:02:56.6277142Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9ab104c2f6083ce.xml 2025-12-04T11:02:56.6549668Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d76d59daf2e605c7.xml 2025-12-04T11:02:56.6842215Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c31c06f41345c5c.xml 2025-12-04T11:02:56.7140651Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a98e316dc116cc1.xml 2025-12-04T11:02:56.7549254Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-723af9187a42bd99.xml 2025-12-04T11:02:56.7826896Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf92d20898f195ab.xml 2025-12-04T11:02:56.8104022Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-524dc1dcdc5a1168.xml 2025-12-04T11:02:56.8380844Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98d2197e9fdc8948.xml 2025-12-04T11:02:56.8639986Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2102f231775de0a.xml 2025-12-04T11:02:56.8900165Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-271cbb7ef1d534a9.xml 2025-12-04T11:02:56.9220889Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d42c353b826e766.xml 2025-12-04T11:02:56.9523841Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4f9face38efde3d.xml 2025-12-04T11:02:56.9802303Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65eb8dabd26f049f.xml 2025-12-04T11:02:57.0088947Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-993c00909ddd3eaa.xml 2025-12-04T11:02:57.0402957Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a20300b26842835f.xml 2025-12-04T11:02:57.0717139Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8300341a4195c4a2.xml 2025-12-04T11:02:57.1000317Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e45287a9f93d9858.xml 2025-12-04T11:02:57.1290341Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa4ae74c761f06da.xml 2025-12-04T11:02:57.1608386Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b7742dab30c5a9.xml 2025-12-04T11:02:57.2001484Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92706bcf8036c15a.xml 2025-12-04T11:02:57.2322243Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e2d29ed0b71edf9.xml 2025-12-04T11:02:57.2612093Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b82693979bf3f797.xml 2025-12-04T11:02:57.2892551Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40f956e1991c429b.xml 2025-12-04T11:02:57.3161925Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be8fbd355ae7d4d2.xml 2025-12-04T11:02:57.3469702Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00fdf0a71d560046.xml 2025-12-04T11:02:57.3729431Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f23f45b0fd1b4e0.xml 2025-12-04T11:02:57.4002975Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a704bee28922d105.xml 2025-12-04T11:02:57.4270358Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59aa1d9ad1b952bc.xml 2025-12-04T11:02:57.4571666Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f46bcad2881001f4.xml 2025-12-04T11:02:57.4869527Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bdf0f48b15d753a.xml 2025-12-04T11:02:57.5127487Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3701eb2bf6e59b5c.xml 2025-12-04T11:02:57.5433958Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4294fcb622b36e12.xml 2025-12-04T11:02:57.5748357Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25f21f2c9cc3f58e.xml 2025-12-04T11:02:57.6022968Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a888dfec83025ca2.xml 2025-12-04T11:02:57.6323103Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a2971cbec516283.xml 2025-12-04T11:02:57.6592477Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2615b1994fa3617.xml 2025-12-04T11:02:57.6882445Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e62634fc743f9964.xml 2025-12-04T11:02:57.7171445Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11b374627b30152b.xml 2025-12-04T11:02:57.7458990Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2922c97c2127de2.xml 2025-12-04T11:02:57.7708403Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25e6eb10dce656e7.xml 2025-12-04T11:02:57.8022948Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c2f928358ecde73.xml 2025-12-04T11:02:57.8310083Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d84b93ba23264de.xml 2025-12-04T11:02:57.8613121Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48132f4775de3019.xml 2025-12-04T11:02:57.8899008Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4552be91feabceb2.xml 2025-12-04T11:02:57.9180818Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6475011a4c42f67b.xml 2025-12-04T11:02:57.9459489Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d893aced2165d7e.xml 2025-12-04T11:02:57.9772917Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b44c71018f3811f.xml 2025-12-04T11:02:58.0149655Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26728a48a74f0db0.xml 2025-12-04T11:02:58.0469990Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9014cdddbced68f.xml 2025-12-04T11:02:58.0720651Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-578fd43f9d90672b.xml 2025-12-04T11:02:58.1013111Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-293b2ba538d5c1ee.xml 2025-12-04T11:02:58.1402309Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea83e8b183ac592d.xml 2025-12-04T11:02:58.1699483Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-436923729a977d14.xml 2025-12-04T11:02:58.1990182Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5f688e692439013.xml 2025-12-04T11:02:58.2449694Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4dc08961f281c036.xml 2025-12-04T11:02:58.2750887Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da7051c1be9f2e2f.xml 2025-12-04T11:02:58.3069935Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6675bf2f877791e.xml 2025-12-04T11:02:58.3341429Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f31832653dbf03f.xml 2025-12-04T11:02:58.3639279Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc72cfdb4feaf9b1.xml 2025-12-04T11:02:58.3926908Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4abfc0edb4e297a.xml 2025-12-04T11:02:58.4399858Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a979b5e962d58cfe.xml 2025-12-04T11:02:58.4922687Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80387dc09d6e68db.xml 2025-12-04T11:02:58.5241009Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec959ea151220e03.xml 2025-12-04T11:02:58.5509635Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a47b279193b674dc.xml 2025-12-04T11:02:58.5788205Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8aed26ab454245f.xml 2025-12-04T11:02:58.6069017Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-498b958aacbfbe1c.xml 2025-12-04T11:02:58.6347631Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ca585512d847c09.xml 2025-12-04T11:02:58.6640124Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-654cb3e44738170f.xml 2025-12-04T11:02:58.7019716Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28d3d63fc6efada0.xml 2025-12-04T11:02:58.7298266Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00ace48f549d318a.xml 2025-12-04T11:02:58.7600535Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-673dc73dcfb18eb0.xml 2025-12-04T11:02:58.7890032Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfd6f3e2d018dce1.xml 2025-12-04T11:02:58.8203735Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-325d701d1891cfcd.xml 2025-12-04T11:02:58.8501905Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bda5b12f5f6c4582.xml 2025-12-04T11:02:58.8808157Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71b41420eeb91960.xml 2025-12-04T11:02:58.9072392Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d12eb62cb70e222b.xml 2025-12-04T11:02:58.9382871Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28e8d5b3cc52ed16.xml 2025-12-04T11:02:58.9909710Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3575bf35b7b1c647.xml 2025-12-04T11:02:59.0201593Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4522094cb3ebf4f.xml 2025-12-04T11:02:59.0482817Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9492f0ab3ccaa218.xml 2025-12-04T11:02:59.0779522Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4010fccffcf7594.xml 2025-12-04T11:02:59.1060937Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ffb1b63fa92eef97.xml 2025-12-04T11:02:59.1338972Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6439db28f2af0685.xml 2025-12-04T11:02:59.1639624Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c2bb33ba5b4e7cc.xml 2025-12-04T11:02:59.1900849Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95dc0d7d8e2b1b6d.xml 2025-12-04T11:02:59.2208345Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cde022d82d2adcaa.xml 2025-12-04T11:02:59.2489210Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3ea27d76bf44331.xml 2025-12-04T11:02:59.2790464Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d08d23ef2c0e694a.xml 2025-12-04T11:02:59.3089187Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-862e9783d4f59986.xml 2025-12-04T11:02:59.3342937Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4662846f57df901f.xml 2025-12-04T11:02:59.3670781Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-201ee2adc01d4dba.xml 2025-12-04T11:02:59.3978148Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-232183083f1e6c18.xml 2025-12-04T11:02:59.4264314Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92d5ea877ee6cee8.xml 2025-12-04T11:02:59.5181786Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e7940eab8840d89.xml 2025-12-04T11:02:59.5549575Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55d55fb078c2b684.xml 2025-12-04T11:02:59.5883415Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a0928e0ccb8a6b10.xml 2025-12-04T11:02:59.6182321Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2664d0ec37e344ff.xml 2025-12-04T11:02:59.6492678Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32cc8afe70b3e7da.xml 2025-12-04T11:02:59.6790754Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d3b40b7260221d.xml 2025-12-04T11:02:59.7128398Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ac3d6524701cbf9.xml 2025-12-04T11:02:59.7429880Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15963edc8117391b.xml 2025-12-04T11:02:59.8009840Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc7ad1ec18a7bb1f.xml 2025-12-04T11:02:59.8309184Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4b332b6754f948d.xml 2025-12-04T11:02:59.8590628Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f36e4bfeb187dab.xml 2025-12-04T11:02:59.8888928Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-046129b2924de1d6.xml 2025-12-04T11:02:59.9261054Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1fb6eb7f4419f2c.xml 2025-12-04T11:02:59.9636654Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb907762912a3d99.xml 2025-12-04T11:02:59.9907513Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39f1cb056fff7a78.xml 2025-12-04T11:03:00.0221674Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-285c65bf39f49db0.xml 2025-12-04T11:03:00.0526621Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e83f111f9b87d25.xml 2025-12-04T11:03:00.0832518Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10563eb7a07575a6.xml 2025-12-04T11:03:00.1122558Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc37255b8c6ba83b.xml 2025-12-04T11:03:00.1428164Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13cdaac161151c73.xml 2025-12-04T11:03:00.1723044Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6fc399e73bd2cfea.xml 2025-12-04T11:03:00.2023330Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c09d245ecad9319.xml 2025-12-04T11:03:00.2296435Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0e3e1dcb90d6929.xml 2025-12-04T11:03:00.2600385Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51c71b398d03a4df.xml 2025-12-04T11:03:00.2889116Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3006a46890f44c5.xml 2025-12-04T11:03:00.3190095Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-267dee8bb7025f13.xml 2025-12-04T11:03:00.3706043Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a7645ce5728017b.xml 2025-12-04T11:03:00.4041518Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-deb9bfbb4bd80c4b.xml 2025-12-04T11:03:00.4480448Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e11a11c30a005f3.xml 2025-12-04T11:03:00.4782249Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c85cce560acdb42f.xml 2025-12-04T11:03:00.5081071Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96d41dfdc3fe6dba.xml 2025-12-04T11:03:00.5498942Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c26119bec135fb9d.xml 2025-12-04T11:03:00.5802917Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-114850235e770dcb.xml 2025-12-04T11:03:00.6381104Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ed8dfc73ef01d6d.xml 2025-12-04T11:03:00.6670131Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49b473d112cc4546.xml 2025-12-04T11:03:00.6927648Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc13b9aff20c070d.xml 2025-12-04T11:03:00.7197164Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04e707708d96db67.xml 2025-12-04T11:03:00.7482434Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4697a0ab5cd20b70.xml 2025-12-04T11:03:00.7773322Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a417a16a93526fa4.xml 2025-12-04T11:03:00.8129012Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ddfeed99e9cdec6.xml 2025-12-04T11:03:00.8427732Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16b81ab28b107f75.xml 2025-12-04T11:03:00.8783612Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cff05e49a4594759.xml 2025-12-04T11:03:00.9083952Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db1fdc0a1e7f6276.xml 2025-12-04T11:03:00.9350930Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cb2bb364c0ae81e.xml 2025-12-04T11:03:00.9661309Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16f5e50845c16298.xml 2025-12-04T11:03:00.9923376Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d791b90d6362f3c0.xml 2025-12-04T11:03:01.0212897Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96506a809a7d9377.xml 2025-12-04T11:03:01.0510610Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-140f4b27a4309298.xml 2025-12-04T11:03:01.0783272Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-541ecd361c5cbd66.xml 2025-12-04T11:03:01.1070158Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0707caf97ad64431.xml 2025-12-04T11:03:01.1379185Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a41a3afe18e29798.xml 2025-12-04T11:03:01.1727703Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-384c7454c26fa694.xml 2025-12-04T11:03:01.2067740Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39a728e06646a1d3.xml 2025-12-04T11:03:01.7137462Z Uploading artifacts took 0.47 seconds 2025-12-04T11:03:01.7138279Z Running distributed/test_distributed_spawn 8/9 ... [2025-12-04 11:03:01.713696][7812.651333387] 2025-12-04T11:03:01.7140615Z Running distributed tests for the test backend with env init_method 2025-12-04T11:03:01.7142395Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:03:01.7146160Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:03:01.714433] 2025-12-04T11:03:05.2983298Z 2025-12-04T11:03:05.2984428Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_e096ad8d19f564e6_.log 2025-12-04T11:03:05.2985499Z Running 0 items in this shard: 2025-12-04T11:03:05.2985718Z 2025-12-04T11:03:05.2986893Z Running distributed tests for the test backend with file init_method 2025-12-04T11:03:05.2988988Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:03:05.2992794Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:03:05.299084] 2025-12-04T11:03:08.8828435Z 2025-12-04T11:03:08.8829549Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_541f6e7caefa0f4f_.log 2025-12-04T11:03:08.8830666Z Running 0 items in this shard: 2025-12-04T11:03:08.8830904Z 2025-12-04T11:03:08.8835869Z Running distributed tests for the mpi backend with env init_method 2025-12-04T11:03:09.0070386Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:03:09.0072407Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:03:09.006976] 2025-12-04T11:03:13.2026808Z 2025-12-04T11:03:13.2028053Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_7e460d66e8af4d5d_.log 2025-12-04T11:03:13.2029139Z Running 0 items in this shard: 2025-12-04T11:03:13.2029513Z Running 0 items in this shard: 2025-12-04T11:03:13.2029848Z Running 0 items in this shard: 2025-12-04T11:03:13.2030055Z 2025-12-04T11:03:13.2032430Z Running distributed tests for the mpi backend with file init_method 2025-12-04T11:03:13.3272442Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:03:13.3276590Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:03:13.327404] 2025-12-04T11:03:17.5400640Z 2025-12-04T11:03:17.5401767Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_015ee03be2a51a26_.log 2025-12-04T11:03:17.5402802Z Running 0 items in this shard: 2025-12-04T11:03:17.5403139Z Running 0 items in this shard: 2025-12-04T11:03:17.5403785Z Running 0 items in this shard: 2025-12-04T11:03:17.5403988Z 2025-12-04T11:03:17.5408113Z Running distributed tests for the nccl backend with env init_method 2025-12-04T11:03:17.5408607Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:03:17.5411206Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:03:17.540906] 2025-12-04T11:06:25.9699645Z 2025-12-04T11:06:25.9700808Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_ebb0a88e8fde4492_.log 2025-12-04T11:06:25.9717198Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-12-04T11:06:25.9732985Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2025-12-04T11:06:25.9734542Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2025-12-04T11:06:25.9736001Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2025-12-04T11:06:25.9737238Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2025-12-04T11:06:25.9738292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2025-12-04T11:06:25.9739509Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-12-04T11:06:25.9740890Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2025-12-04T11:06:25.9742283Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2025-12-04T11:06:25.9743566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-12-04T11:06:25.9744693Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2025-12-04T11:06:25.9745932Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2025-12-04T11:06:25.9747260Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2025-12-04T11:06:25.9748824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2025-12-04T11:06:25.9750439Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-12-04T11:06:25.9751928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2025-12-04T11:06:25.9753426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2025-12-04T11:06:25.9754648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-12-04T11:06:25.9756017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2025-12-04T11:06:25.9757347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2025-12-04T11:06:25.9758784Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-12-04T11:06:25.9760055Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2025-12-04T11:06:25.9761229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2025-12-04T11:06:25.9762345Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2025-12-04T11:06:25.9763505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2025-12-04T11:06:25.9764780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-12-04T11:06:25.9765945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-12-04T11:06:25.9766568Z 2025-12-04T11:06:25.9766820Z Running distributed tests for the nccl backend with file init_method 2025-12-04T11:06:25.9767343Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:06:25.9768899Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:06:25.971326] 2025-12-04T11:09:34.3026463Z 2025-12-04T11:09:34.3027743Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_4d339c4268e751fc_.log 2025-12-04T11:09:34.3043256Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-12-04T11:09:34.3058137Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2025-12-04T11:09:34.3059729Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2025-12-04T11:09:34.3061069Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2025-12-04T11:09:34.3062214Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2025-12-04T11:09:34.3063374Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2025-12-04T11:09:34.3064588Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-12-04T11:09:34.3065936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2025-12-04T11:09:34.3067432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2025-12-04T11:09:34.3068945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-12-04T11:09:34.3070116Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2025-12-04T11:09:34.3071383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2025-12-04T11:09:34.3072688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2025-12-04T11:09:34.3074080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2025-12-04T11:09:34.3075705Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-12-04T11:09:34.3077181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2025-12-04T11:09:34.3078453Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2025-12-04T11:09:34.3079787Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-12-04T11:09:34.3081013Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2025-12-04T11:09:34.3082286Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2025-12-04T11:09:34.3083620Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-12-04T11:09:34.3084857Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2025-12-04T11:09:34.3085991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2025-12-04T11:09:34.3087089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2025-12-04T11:09:34.3088189Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2025-12-04T11:09:34.3089346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-12-04T11:09:34.3090476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-12-04T11:09:34.3091112Z 2025-12-04T11:09:34.3091365Z Running distributed tests for the gloo backend with env init_method 2025-12-04T11:09:34.3091849Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:09:34.3093170Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:09:34.304273] 2025-12-04T11:13:03.0916686Z 2025-12-04T11:13:03.0917794Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_2be7bbd7c30a5d09_.log 2025-12-04T11:13:03.0933690Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-12-04T11:13:03.0949023Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2025-12-04T11:13:03.0950562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2025-12-04T11:13:03.0952012Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2025-12-04T11:13:03.0953182Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2025-12-04T11:13:03.0954287Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2025-12-04T11:13:03.0955518Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-12-04T11:13:03.0956990Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2025-12-04T11:13:03.0958430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2025-12-04T11:13:03.0959899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-12-04T11:13:03.0961032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2025-12-04T11:13:03.0962252Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2025-12-04T11:13:03.0963514Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2025-12-04T11:13:03.0964863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2025-12-04T11:13:03.0966431Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-12-04T11:13:03.0967871Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2025-12-04T11:13:03.0969114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2025-12-04T11:13:03.0970286Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-12-04T11:13:03.0971516Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2025-12-04T11:13:03.0972800Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2025-12-04T11:13:03.0974061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-12-04T11:13:03.0975367Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2025-12-04T11:13:03.0976501Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2025-12-04T11:13:03.0977598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2025-12-04T11:13:03.0978704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2025-12-04T11:13:03.0979873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-12-04T11:13:03.0981006Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-12-04T11:13:03.0981613Z 2025-12-04T11:13:03.0981873Z Running distributed tests for the gloo backend with file init_method 2025-12-04T11:13:03.0982398Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:13:03.0983720Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:13:03.093477] 2025-12-04T11:16:31.8813477Z 2025-12-04T11:16:31.8814539Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_37715603be2e29d4_.log 2025-12-04T11:16:31.8830415Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-12-04T11:16:31.8845591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2025-12-04T11:16:31.8847091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2025-12-04T11:16:31.8848496Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2025-12-04T11:16:31.8849644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2025-12-04T11:16:31.8850794Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2025-12-04T11:16:31.8852004Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-12-04T11:16:31.8853354Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2025-12-04T11:16:31.8854820Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2025-12-04T11:16:31.8856107Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-12-04T11:16:31.8857229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2025-12-04T11:16:31.8858483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2025-12-04T11:16:31.8859743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2025-12-04T11:16:31.8861093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2025-12-04T11:16:31.8862653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-12-04T11:16:31.8864081Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2025-12-04T11:16:31.8865327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2025-12-04T11:16:31.8866503Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-12-04T11:16:31.8867977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2025-12-04T11:16:31.8869298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2025-12-04T11:16:31.8870614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-12-04T11:16:31.8871887Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2025-12-04T11:16:31.8873048Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2025-12-04T11:16:31.8874246Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2025-12-04T11:16:31.8875397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2025-12-04T11:16:31.8876599Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2025-12-04T11:16:31.8877746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2025-12-04T11:16:31.8878395Z 2025-12-04T11:16:31.8878803Z Finished distributed/test_distributed_spawn 8/9 ... [2025-12-04 11:16:31.882039][8622.819673152], took 13.50min 2025-12-04T11:16:31.8998201Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6f6e95ed9bc43f3.xml 2025-12-04T11:16:31.9776890Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17e902b4247dcc9d.xml 2025-12-04T11:16:32.0005411Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a3b3a8e58fbb071.xml 2025-12-04T11:16:32.0256718Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c18a85b55f10e02.xml 2025-12-04T11:16:32.0511007Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee7c4f0b0fe742e4.xml 2025-12-04T11:16:32.0745010Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a74d4ada0231688.xml 2025-12-04T11:16:32.1072194Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-831b908a573477b4.xml 2025-12-04T11:16:32.1375504Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-838476cdd85e63de.xml 2025-12-04T11:16:32.1673851Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7c1e5f9052a62ed.xml 2025-12-04T11:16:32.1967596Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9fe4918eb2ec3fc.xml 2025-12-04T11:16:32.2298068Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18ee4fc76444d2ea.xml 2025-12-04T11:16:32.2640165Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf306b33274dddf6.xml 2025-12-04T11:16:32.3120661Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c91a26250cab8d8.xml 2025-12-04T11:16:32.3396849Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12d1ac9616f3d37b.xml 2025-12-04T11:16:32.4087449Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24baa892d95df88d.xml 2025-12-04T11:16:32.4509257Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1238a6b3de9abf1.xml 2025-12-04T11:16:32.5046937Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5960ed309c5523d7.xml 2025-12-04T11:16:32.5378429Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-484220487412d630.xml 2025-12-04T11:16:32.5786355Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcdabce46aafa3d7.xml 2025-12-04T11:16:32.6240135Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-712b9444633e7909.xml 2025-12-04T11:16:32.6605786Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22f63a5524b50ec1.xml 2025-12-04T11:16:32.6976290Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9abdeaa3f668748b.xml 2025-12-04T11:16:32.7390835Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db023d74613c38df.xml 2025-12-04T11:16:32.7793339Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d349e5cd46aba4b9.xml 2025-12-04T11:16:32.8217947Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dcab0e823bbcb484.xml 2025-12-04T11:16:32.8568236Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-107117b39386c513.xml 2025-12-04T11:16:32.9119290Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2a16f55ac9cd547.xml 2025-12-04T11:16:32.9549612Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f920c9d47a13120.xml 2025-12-04T11:16:33.0108919Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0adedc6075462154.xml 2025-12-04T11:16:33.0498625Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51b3a60aa7e4ef26.xml 2025-12-04T11:16:33.0911963Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9abab74452896355.xml 2025-12-04T11:16:33.1326365Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa158d78c6a1047c.xml 2025-12-04T11:16:33.1789185Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa80501c9f66d90c.xml 2025-12-04T11:16:33.2160833Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e0dbc7fa11b6547.xml 2025-12-04T11:16:33.2565536Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75e1b023ce49c02c.xml 2025-12-04T11:16:33.2876914Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09a1c22c519254b6.xml 2025-12-04T11:16:33.3272162Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c214051d59b82c2.xml 2025-12-04T11:16:33.3578839Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed7ddfe1482b6f0b.xml 2025-12-04T11:16:33.3910325Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a95a417b4eefa26.xml 2025-12-04T11:16:33.4187674Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2ef4a32ca2b3dbf.xml 2025-12-04T11:16:33.4505542Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5c9327604cb8f14.xml 2025-12-04T11:16:33.4832931Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c38e0bbe495b143b.xml 2025-12-04T11:16:33.5144712Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cb5b2f78eac5bfd.xml 2025-12-04T11:16:33.5434119Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f44a2b1c1e232df9.xml 2025-12-04T11:16:33.5744677Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7f8edc520bdf516e.xml 2025-12-04T11:16:33.6014677Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-402e1eebb6506e23.xml 2025-12-04T11:16:33.6327034Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d8076bdf0a69595.xml 2025-12-04T11:16:33.6711965Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b468bab3124b13b3.xml 2025-12-04T11:16:33.7195867Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e50d3219cb7bfbb6.xml 2025-12-04T11:16:33.7463218Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53454fb25c79a584.xml 2025-12-04T11:16:33.7751957Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ee8c69fe7ba0518.xml 2025-12-04T11:16:33.8074372Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1220c9553e1d6abf.xml 2025-12-04T11:16:33.8373949Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f13fc2d140babf7f.xml 2025-12-04T11:16:33.8695864Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-557712f590042c69.xml 2025-12-04T11:16:33.9063838Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce76f7d0c0c1d9a9.xml 2025-12-04T11:16:33.9391749Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-323c51dd40cd5ff1.xml 2025-12-04T11:16:33.9655783Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fbb3e53e76b1f52.xml 2025-12-04T11:16:34.0026631Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17f25234f4eef787.xml 2025-12-04T11:16:34.0386798Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98105ea07b2795a4.xml 2025-12-04T11:16:34.0669561Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecac8ed92cb8e9f2.xml 2025-12-04T11:16:34.1073866Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abee6adb87748222.xml 2025-12-04T11:16:34.1378197Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0c43579cac56c01.xml 2025-12-04T11:16:34.1938336Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-838afcd07f98645e.xml 2025-12-04T11:16:34.2254594Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-efef7c0c73e4a3a4.xml 2025-12-04T11:16:34.2598226Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-279a5af71a085b78.xml 2025-12-04T11:16:34.2869027Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03480c4b462e8e3c.xml 2025-12-04T11:16:34.3506765Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32560068724b69d2.xml 2025-12-04T11:16:34.3846680Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e0917bfdcbaebaa.xml 2025-12-04T11:16:34.4400486Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-895373b2e146bc0e.xml 2025-12-04T11:16:34.5109879Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c7263c0c2b97dd5.xml 2025-12-04T11:16:34.5525458Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-607d80da638aa01c.xml 2025-12-04T11:16:34.5873989Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-328e7d3fc5b20fc8.xml 2025-12-04T11:16:34.6538770Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3fc9d94de1faf069.xml 2025-12-04T11:16:34.6898341Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-725b83e279a2fc0a.xml 2025-12-04T11:16:34.7184513Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd5d50730f4030d9.xml 2025-12-04T11:16:34.7576097Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d65188606bc4fb38.xml 2025-12-04T11:16:34.7854404Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-118492c3de10278a.xml 2025-12-04T11:16:34.8151920Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-892e44a3547470c5.xml 2025-12-04T11:16:34.8457911Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c889ae266192a3d2.xml 2025-12-04T11:16:34.8850845Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91060a4136f305a0.xml 2025-12-04T11:16:34.9156225Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abd0041fabca6f79.xml 2025-12-04T11:16:34.9494493Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28c8471f94c139a5.xml 2025-12-04T11:16:34.9864277Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-794deb0b1425e8fa.xml 2025-12-04T11:16:35.0254430Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d2d9e77b4c2f3d8.xml 2025-12-04T11:16:35.0678507Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ddaf75327e52f1c1.xml 2025-12-04T11:16:35.0992017Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ec25664c586fadf.xml 2025-12-04T11:16:35.1298816Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08df3d74d553659c.xml 2025-12-04T11:16:35.1633101Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d31b0daadd5824cb.xml 2025-12-04T11:16:35.1977477Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63eea3de02c3a6ef.xml 2025-12-04T11:16:35.2336665Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-656d9998e6c222a4.xml 2025-12-04T11:16:35.2688882Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6acaa860eb966f60.xml 2025-12-04T11:16:35.2979183Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce80904a0746c1f8.xml 2025-12-04T11:16:35.3295228Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13fade6629382363.xml 2025-12-04T11:16:35.3597494Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c48a0039a0795ab.xml 2025-12-04T11:16:35.3923281Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1960e6db1730a4a3.xml 2025-12-04T11:16:35.4660228Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b68267f25edfed92.xml 2025-12-04T11:16:35.5058210Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-011a1d9cca18ee04.xml 2025-12-04T11:16:35.5392543Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e9ae608f7e37be8.xml 2025-12-04T11:16:35.5691104Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07300589f39fc3ac.xml 2025-12-04T11:16:35.5991428Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cbc9db131dce2e6.xml 2025-12-04T11:16:35.6406479Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-458e56e213b7ad87.xml 2025-12-04T11:16:35.7400637Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d87679cc1c529e2d.xml 2025-12-04T11:16:35.7686962Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4b08dbbb6915206.xml 2025-12-04T11:16:35.8040806Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30aed6037f140bb7.xml 2025-12-04T11:16:35.8410568Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-689b3f622d74137a.xml 2025-12-04T11:16:35.8785616Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cb150e7fa6adefe.xml 2025-12-04T11:16:35.9138909Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-773bb593cee0da12.xml 2025-12-04T11:16:35.9453234Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3c6b8d3271f370c.xml 2025-12-04T11:16:35.9740054Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7116fa9f879b5f5c.xml 2025-12-04T11:16:36.0352099Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab516b6bd9172a5a.xml 2025-12-04T11:16:36.0709292Z Running distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2025-12-04 11:16:36.070282][8627.007919212] 2025-12-04T11:16:36.0709992Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:16:36.0711348Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:16:36.070673] 2025-12-04T11:16:40.3464592Z 2025-12-04T11:16:40.3465865Z distributed/fsdp/test_checkpoint_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_808cfbd3f4bab105_.log 2025-12-04T11:16:40.3471942Z Running 8 items in this shard: test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_apply_activation_checkpointing, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_args_kwargs, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_cpu_offload, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_kwarg_support, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_parity, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_forward_missing_attributes, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_fqn, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_load_activation_checkpointed_module 2025-12-04T11:16:40.3476683Z 2025-12-04T11:16:40.3477122Z Finished distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2025-12-04 11:16:40.345899][8631.28353401], took 0.07min 2025-12-04T11:16:40.3627552Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-e79a881124a4ef8b.xml 2025-12-04T11:16:40.4595741Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2025-12-04 11:16:40.459345][8631.396982249] 2025-12-04T11:16:40.4596483Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:16:40.4598676Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:16:40.459676] 2025-12-04T11:17:24.8376043Z 2025-12-04T11:17:24.8379853Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_a57a4bd8eeed9ab0_.log 2025-12-04T11:17:24.8388330Z Running 9 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions0, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions1, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions2, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithCaching::test_read_write_shard_tensor 2025-12-04T11:17:24.8395389Z 2025-12-04T11:17:24.8395860Z Finished distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2025-12-04 11:17:24.837166][8675.774800218], took 0.74min 2025-12-04T11:17:24.8546775Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-8e7248bfdb62ae56.xml 2025-12-04T11:17:24.9909274Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2025-12-04 11:17:24.990291][8675.927928678] 2025-12-04T11:17:24.9910226Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:17:24.9911642Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:17:24.990645] 2025-12-04T11:18:10.0186331Z 2025-12-04T11:18:10.0188020Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_858ee0662a857cc8_.log 2025-12-04T11:18:10.0201693Z Running 16 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadRot13::test_read_write_tensor_and_blob_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadRot13::test_read_write_tensor_and_blob_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadZStandard::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadZStandard::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2 2025-12-04T11:18:10.0214198Z 2025-12-04T11:18:10.0214873Z Finished distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2025-12-04 11:18:10.018314][8720.955949344], took 0.75min 2025-12-04T11:18:10.0368525Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint_cpu/distributed.checkpoint.test_file_system_checkpoint_cpu-86828f53aae09692.xml 2025-12-04T11:18:10.1462250Z Running distributed/fsdp/test_fsdp_comm_hooks 1/1 ... [2025-12-04 11:18:10.145652][8721.083289594] 2025-12-04T11:18:10.1462896Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:18:10.1464173Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:18:10.145988] 2025-12-04T11:21:17.1897544Z 2025-12-04T11:21:17.1898708Z distributed/fsdp/test_fsdp_comm_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_324153037e83ebec_.log 2025-12-04T11:21:17.1918480Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_hybrid_strategy, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy2 2025-12-04T11:21:17.1937749Z 2025-12-04T11:21:17.1938155Z Finished distributed/fsdp/test_fsdp_comm_hooks 1/1 ... [2025-12-04 11:21:17.189562][8908.127197369], took 3.12min 2025-12-04T11:21:17.2070145Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm_hooks/distributed.fsdp.test_fsdp_comm_hooks-e30f0f398ca36b1a.xml 2025-12-04T11:21:17.3235829Z Running distributed/_shard/test_sharder 1/1 ... [2025-12-04 11:21:17.323352][8908.260988332] 2025-12-04T11:21:17.3236456Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:21:17.3238680Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:21:17.323680] 2025-12-04T11:21:33.1291330Z 2025-12-04T11:21:33.1292499Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_03b9a7c4139e541b_.log 2025-12-04T11:21:33.1294282Z Running 2 items in this shard: test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder, test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder_errors 2025-12-04T11:21:33.1295379Z 2025-12-04T11:21:33.1295743Z Finished distributed/_shard/test_sharder 1/1 ... [2025-12-04 11:21:33.128611][8924.066245576], took 0.26min 2025-12-04T11:21:33.1460226Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._shard.test_sharder/distributed._shard.test_sharder-c376df3131abbe36.xml 2025-12-04T11:21:33.2749030Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2025-12-04 11:21:33.274196][8924.211832813] 2025-12-04T11:21:33.2749762Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:21:33.2751162Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:21:33.274543] 2025-12-04T11:22:06.6248829Z 2025-12-04T11:22:06.6252253Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_01149ea100987574_.log 2025-12-04T11:22:06.6255661Z Running 5 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_clone, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_deep_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_detach, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_inplace_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_set_requires_grad 2025-12-04T11:22:06.6258118Z 2025-12-04T11:22:06.6258593Z Finished distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2025-12-04 11:22:06.624445][8957.56208097], took 0.56min 2025-12-04T11:22:06.6422558Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-130017345592955f.xml 2025-12-04T11:22:06.8320185Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2025-12-04 11:22:06.831777][8957.769413744] 2025-12-04T11:22:06.8320845Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:22:06.8323260Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:22:06.832122] 2025-12-04T11:22:34.3209217Z 2025-12-04T11:22:34.3212111Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_5e35faa3f8e40e8a_.log 2025-12-04T11:22:34.3215692Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state 2025-12-04T11:22:34.3217391Z 2025-12-04T11:22:34.3217817Z Finished distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2025-12-04 11:22:34.320405][8985.258041265], took 0.46min 2025-12-04T11:22:34.3383871Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-4367c9892b5119af.xml 2025-12-04T11:22:34.4851782Z Running distributed/test_c10d_pypg 1/1 ... [2025-12-04 11:22:34.484728][8985.422364937] 2025-12-04T11:22:34.4852410Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:22:34.4853792Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:22:34.485073] 2025-12-04T11:22:41.1159756Z 2025-12-04T11:22:41.1160839Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_264bca3f92d10a9e_.log 2025-12-04T11:22:41.1185452Z Running 48 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_no_init_sync, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_no_init_sync, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_abort_shutdown, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_attr_overrides, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_block_current_stream, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_block_current_stream_use_after_free 2025-12-04T11:22:41.1209978Z 2025-12-04T11:22:41.1210324Z Finished distributed/test_c10d_pypg 1/1 ... [2025-12-04 11:22:41.115443][8992.053076957], took 0.11min 2025-12-04T11:22:41.1333284Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-57a88f260c958571.xml 2025-12-04T11:22:41.2522559Z Running distributed/test_pg_wrapper 1/1 ... [2025-12-04 11:22:41.251828][8992.189465289] 2025-12-04T11:22:41.2523152Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:22:41.2525064Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:22:41.252163] 2025-12-04T11:25:15.6888112Z 2025-12-04T11:25:15.6889127Z distributed/test_pg_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_pg_wrapper_1.1_ebb877765db6c2f1_.log 2025-12-04T11:25:15.6898766Z Running 17 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_debug_level_detail_no_gloo, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_new_group_no_gloo, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2025-12-04T11:25:15.6909047Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail 2025-12-04T11:25:15.6910261Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang 2025-12-04T11:25:15.6911503Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail 2025-12-04T11:25:15.6912874Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off 2025-12-04T11:25:15.6914165Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch 2025-12-04T11:25:15.6915400Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode 2025-12-04T11:25:15.6916652Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_debug_level_detail_no_gloo 2025-12-04T11:25:15.6917806Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_new_group_no_gloo 2025-12-04T11:25:15.6918906Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang 2025-12-04T11:25:15.6920161Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda 2025-12-04T11:25:15.6921436Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode 2025-12-04T11:25:15.6922873Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode 2025-12-04T11:25:15.6924561Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off 2025-12-04T11:25:15.6925826Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch 2025-12-04T11:25:15.6927037Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda 2025-12-04T11:25:15.6928339Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode 2025-12-04T11:25:15.6929670Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2025-12-04T11:25:15.6930383Z 2025-12-04T11:25:15.6930742Z Finished distributed/test_pg_wrapper 1/1 ... [2025-12-04 11:25:15.688701][9146.62633581], took 2.57min 2025-12-04T11:25:15.7069661Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5c2225b883027b57.xml 2025-12-04T11:25:15.7820301Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e989bf90bb499159.xml 2025-12-04T11:25:15.8125703Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5358d199d91e916e.xml 2025-12-04T11:25:15.8427424Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e8f01c1258f32697.xml 2025-12-04T11:25:15.8818517Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-ac92f4ba57cf76c4.xml 2025-12-04T11:25:15.9113015Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-eda4e22775599b5a.xml 2025-12-04T11:25:15.9426453Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-f2042aa2d074e648.xml 2025-12-04T11:25:15.9721727Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-eace822c5c4d90b0.xml 2025-12-04T11:25:16.0018932Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e2aaf0e6db1856f6.xml 2025-12-04T11:25:16.0322538Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-aba1e98d5f917e6f.xml 2025-12-04T11:25:16.0640545Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-c8752499cbdebb6b.xml 2025-12-04T11:25:16.0914704Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e365ef3c3de4e526.xml 2025-12-04T11:25:16.1472208Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-3d0c386b38968978.xml 2025-12-04T11:25:16.1880612Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-c9893cb7ff9eb7cb.xml 2025-12-04T11:25:16.2177556Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-d8ca55d72dd5d963.xml 2025-12-04T11:25:16.2488627Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-d6cc878cd0b0d21e.xml 2025-12-04T11:25:16.2798322Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5399ee4967a63bf6.xml 2025-12-04T11:25:16.7742194Z Uploading artifacts took 0.42 seconds 2025-12-04T11:25:16.7743042Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2025-12-04 11:25:16.774115][9147.711751315] 2025-12-04T11:25:16.7743720Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:25:16.7746582Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:25:16.774459] 2025-12-04T11:25:20.5007534Z 2025-12-04T11:25:20.5009069Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_cd3116aa947c2691_.log 2025-12-04T11:25:20.5010198Z Running 0 items in this shard: 2025-12-04T11:25:20.5010423Z 2025-12-04T11:25:20.5010860Z Finished distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2025-12-04 11:25:20.500160][9151.43779699], took 0.06min 2025-12-04T11:25:20.5195178Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-97a90e2322241ed5.xml 2025-12-04T11:25:20.5873698Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 ... [2025-12-04 11:25:20.587140][9151.524776399] 2025-12-04T11:25:20.5874427Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:25:20.5876634Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:25:20.587468] 2025-12-04T11:26:02.2551929Z 2025-12-04T11:26:02.2553432Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_49f5842fb075f2b2_.log 2025-12-04T11:26:02.2557052Z Running 4 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose_tensor_specs, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal_tensor_specs 2025-12-04T11:26:02.2559550Z 2025-12-04T11:26:02.2560147Z Finished distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 ... [2025-12-04 11:26:02.255227][9193.192861024], took 0.69min 2025-12-04T11:26:02.2750978Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_binary_cmp/distributed._shard.sharded_tensor.ops.test_binary_cmp-1b60820ad9df53ea.xml 2025-12-04T11:26:02.4122358Z Running distributed/nn/jit/test_instantiator 1/1 ... [2025-12-04 11:26:02.411842][9193.349479111] 2025-12-04T11:26:02.4123020Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:26:02.4125676Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:26:02.412179] 2025-12-04T11:26:06.2366891Z 2025-12-04T11:26:06.2368408Z distributed/nn/jit/test_instantiator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.nn.jit.test_instantiator_1.1_48df896d98c27f75_.log 2025-12-04T11:26:06.2371212Z Running 3 items in this shard: test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_get_arg_return_types_from_interface, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_non_scripted_remote_module_template, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_scripted_remote_module_template 2025-12-04T11:26:06.2373166Z 2025-12-04T11:26:06.2373679Z Finished distributed/nn/jit/test_instantiator 1/1 ... [2025-12-04 11:26:06.236136][9197.173771738], took 0.06min 2025-12-04T11:26:06.2555748Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.nn.jit.test_instantiator/distributed.nn.jit.test_instantiator-26bb23415f1ce435.xml 2025-12-04T11:26:06.3315758Z Running distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2025-12-04 11:26:06.331353][9197.268988142] 2025-12-04T11:26:06.3316664Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:26:06.3318865Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:26:06.331679] 2025-12-04T11:26:33.9146113Z 2025-12-04T11:26:33.9149935Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_60479538b31460df_.log 2025-12-04T11:26:33.9157154Z Running 11 items in this shard: test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_check_overlapping, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_chunked_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_device_placement, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_enumerable_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunk_sharding_params, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunked_dim_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_split_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_infer_sharding_spec_from_shards_metadata, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_shard_tensor, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_tensor_ctor 2025-12-04T11:26:33.9163423Z 2025-12-04T11:26:33.9163902Z Finished distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2025-12-04 11:26:33.914028][9224.851663896], took 0.46min 2025-12-04T11:26:33.9343887Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._shard.sharding_spec.test_sharding_spec/distributed._shard.sharding_spec.test_sharding_spec-b7677c73b05bb97d.xml 2025-12-04T11:26:34.0503566Z Running distributed/test_nccl 1/1 ... [2025-12-04 11:26:34.049864][9224.987500258] 2025-12-04T11:26:34.0504124Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:26:34.0505323Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:26:34.050212] 2025-12-04T11:26:40.1306808Z 2025-12-04T11:26:40.1308661Z distributed/test_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nccl_1.1_6ae7ff7c670cceb0_.log 2025-12-04T11:26:40.1314412Z Running 13 items in this shard: test/distributed/test_nccl.py::NCCLSymmetricMemoryTest::test_nccl_symmem_alloc, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float8_e4m3fn, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float8_e5m2, test/distributed/test_nccl.py::TestNCCLCUDA::test_collective_errors_cuda, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_unique_id_cuda 2025-12-04T11:26:40.1318846Z 2025-12-04T11:26:40.1319159Z Finished distributed/test_nccl 1/1 ... [2025-12-04 11:26:40.130173][9231.067808625], took 0.10min 2025-12-04T11:26:40.1505008Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_nccl/distributed.test_nccl-6ab9552fe0bfe15f.xml 2025-12-04T11:26:40.2767001Z Running distributed/fsdp/test_fsdp_misc 1/1 ... [2025-12-04 11:26:40.276132][9231.213769309] 2025-12-04T11:26:40.2767798Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:26:40.2769044Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:26:40.276477] 2025-12-04T11:27:54.1820838Z 2025-12-04T11:27:54.1821910Z distributed/fsdp/test_fsdp_misc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_91e245634b3e3a2d_.log 2025-12-04T11:27:54.1838299Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_cpu_init_with_sync_module_states, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_init_stays_on_cpu, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_training, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_False, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_True, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_not_all_outputs_used_in_loss, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optim_overlap_no_use_orig_params_error, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optimizer_overlap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_zero2_eval_with_prefetch, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_cpu_gpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_device_id_auto_wrap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_cpu_offload, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_no_move_ignored_params_and_bufs, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_ignored_module_meta, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_namedtuple, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_same_model_across_ranks, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_unsupported_module_cls, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_homogeneous_attributes, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_module_device_mismatches_device_id, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_multigpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_no_params, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_training_device_mismatch_errors, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_unsafe_setattr, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_world_size_1_sharding_strategy_warning 2025-12-04T11:27:54.1852563Z 2025-12-04T11:27:54.1852944Z Finished distributed/fsdp/test_fsdp_misc 1/1 ... [2025-12-04 11:27:54.181601][9305.119235909], took 1.23min 2025-12-04T11:27:54.2024618Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_misc/distributed.fsdp.test_fsdp_misc-ff29f8b606fc4847.xml 2025-12-04T11:27:54.3333872Z Running distributed/fsdp/test_fsdp_meta 1/1 ... [2025-12-04 11:27:54.333166][9305.270802985] 2025-12-04T11:27:54.3334683Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:27:54.3336900Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:27:54.333492] 2025-12-04T11:28:38.1608191Z 2025-12-04T11:28:38.1609287Z distributed/fsdp/test_fsdp_meta 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_d514c850fca6a88f_.log 2025-12-04T11:28:38.1621540Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_meta, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_torchdistx, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_meta_device_with_mixed_precision, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_reset_params, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_init_fn 2025-12-04T11:28:38.1630911Z 2025-12-04T11:28:38.1631625Z Finished distributed/fsdp/test_fsdp_meta 1/1 ... [2025-12-04 11:28:38.160459][9349.098094483], took 0.73min 2025-12-04T11:28:38.1814473Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_meta/distributed.fsdp.test_fsdp_meta-8050e6a6d02fe535.xml 2025-12-04T11:28:38.2974363Z Running distributed/test_data_parallel 1/1 ... [2025-12-04 11:28:38.297028][9349.234665317] 2025-12-04T11:28:38.2975029Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:28:38.2976278Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:28:38.297345] 2025-12-04T11:28:47.4850222Z 2025-12-04T11:28:47.4851742Z distributed/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_data_parallel_1.1_917049db61ce8799_.log 2025-12-04T11:28:47.4875765Z Running 46 items in this shard: test/distributed/test_data_parallel.py::TestDataParallel::test_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_buffers_requiring_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_complex, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_device_args, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_function_deletion, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_lazy_linear, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_device, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_no_refcycles, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_module_zero_inputs, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_multiple_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_output, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_no_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_rnn, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_small_back, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_sparse, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_different_len_dicts, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_passes_exception, test/distributed/test_data_parallel.py::TestDataParallel::test_parameter_list_dict_replica, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate_buffers, test/distributed/test_data_parallel.py::TestDataParallel::test_save_replica_module, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_strided_grad_layout, test/distributed/test_data_parallel.py::TestDataParallel::test_zero_grad, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 2025-12-04T11:28:47.4896474Z 2025-12-04T11:28:47.4896812Z Finished distributed/test_data_parallel 1/1 ... [2025-12-04 11:28:47.484456][9358.422090966], took 0.15min 2025-12-04T11:28:47.5053931Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_data_parallel/distributed.test_data_parallel-c1f5c0fcbf415006.xml 2025-12-04T11:28:47.6513736Z Running distributed/rpc/cuda/test_tensorpipe_agent 2/2 ... [2025-12-04 11:28:47.651081][9358.58871788] 2025-12-04T11:28:47.6514426Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:28:47.6516563Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '--shard-id=2', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:28:47.651408] 2025-12-04T11:39:11.9777915Z 2025-12-04T11:39:11.9779077Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_2.2_022525b628906125_.log 2025-12-04T11:39:11.9815249Z Running 52 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value, test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass 2025-12-04T11:39:11.9851155Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device 2025-12-04T11:39:11.9852612Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeCudaRemoteModuleTest::test_input_moved_to_cuda_device_script 2025-12-04T11:39:11.9854214Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_async_execution_nested_with_cuda_future 2025-12-04T11:39:11.9855788Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_callback_changes_devices 2025-12-04T11:39:11.9857383Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_cuda_sparse_tensor 2025-12-04T11:39:11.9859094Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor 2025-12-04T11:39:11.9860848Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_can_extract_custom_class_with_cuda_tensor 2025-12-04T11:39:11.9862449Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_device 2025-12-04T11:39:11.9863900Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_int 2025-12-04T11:39:11.9865354Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_device_as_str 2025-12-04T11:39:11.9866851Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_modify_tensor_inplace 2025-12-04T11:39:11.9868632Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_replace_tensor 2025-12-04T11:39:11.9870183Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_cuda_future_value_on_bad_device 2025-12-04T11:39:11.9871643Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream 2025-12-04T11:39:11.9873054Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_multi 2025-12-04T11:39:11.9874656Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_custom_stream_nested_multi 2025-12-04T11:39:11.9876189Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_default 2025-12-04T11:39:11.9877747Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_cpu_to_gpu_non_default 2025-12-04T11:39:11.9879404Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_7 2025-12-04T11:39:11.9880907Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_8 2025-12-04T11:39:11.9882313Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_1 2025-12-04T11:39:11.9883775Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_mixed_self_7 2025-12-04T11:39:11.9885300Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_non_default 2025-12-04T11:39:11.9886656Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_map_gpu_to_cpu_default 2025-12-04T11:39:11.9887993Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_gpu 2025-12-04T11:39:11.9889327Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_local_device 2025-12-04T11:39:11.9890757Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_invalid_max_remote_device 2025-12-04T11:39:11.9892135Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_many_to_one 2025-12-04T11:39:11.9893455Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config 2025-12-04T11:39:11.9894826Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_loop 2025-12-04T11:39:11.9896234Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_not_timeout 2025-12-04T11:39:11.9897702Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_remote_response 2025-12-04T11:39:11.9899158Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_missing_config_response 2025-12-04T11:39:11.9900514Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu 2025-12-04T11:39:11.9901832Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_multi_gpu_self 2025-12-04T11:39:11.9903118Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_device_maps_remote 2025-12-04T11:39:11.9904406Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_devices_option_mismatch 2025-12-04T11:39:11.9905775Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization2 2025-12-04T11:39:11.9907255Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_owner_rref_forward_synchronization4 2025-12-04T11:39:11.9908993Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization1 2025-12-04T11:39:11.9910529Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization2 2025-12-04T11:39:11.9912068Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization3 2025-12-04T11:39:11.9913602Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization4 2025-12-04T11:39:11.9915201Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_as_arg_synchronization5 2025-12-04T11:39:11.9916726Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization3 2025-12-04T11:39:11.9918277Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_forward_synchronization4 2025-12-04T11:39:11.9919955Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization1 2025-12-04T11:39:11.9921516Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization2 2025-12-04T11:39:11.9922958Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_to_here_synchronization4 2025-12-04T11:39:11.9924980Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_rref_with_unpickleable_attributes 2025-12-04T11:39:11.9926535Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeAgentCudaRpcTest::test_tensor_view_as_return_value 2025-12-04T11:39:11.9928074Z Running 1 items in this shard: test/distributed/rpc/cuda/test_tensorpipe_agent.py::TensorPipeTensorPipeCudaDistAutogradTest::test_device_maps_backward_pass 2025-12-04T11:39:11.9928916Z 2025-12-04T11:39:11.9929382Z Finished distributed/rpc/cuda/test_tensorpipe_agent 2/2 ... [2025-12-04 11:39:11.979556][9982.917186539], took 10.41min 2025-12-04T11:39:12.0011051Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e8d73f4591d62db9.xml 2025-12-04T11:39:12.1046519Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9a221c145b1a74c2.xml 2025-12-04T11:39:12.1312904Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-08cb4ec451a35d57.xml 2025-12-04T11:39:12.1841253Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dee1ac91d0c754d6.xml 2025-12-04T11:39:12.2159110Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-350598816b45384d.xml 2025-12-04T11:39:12.2711235Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-510c0213c3d43ce4.xml 2025-12-04T11:39:12.3128545Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b9f09d55835d87af.xml 2025-12-04T11:39:12.3390900Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ed820609834e1378.xml 2025-12-04T11:39:12.3764254Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c88e834a8b4dbb20.xml 2025-12-04T11:39:12.4096574Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f0b18dfb3a59dad1.xml 2025-12-04T11:39:12.4485223Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4743d357d297f6dc.xml 2025-12-04T11:39:12.4804332Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4662c39cb6a4b928.xml 2025-12-04T11:39:12.5095691Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-73028dd94dff8c49.xml 2025-12-04T11:39:12.5381537Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3ee7cb942a2cc6e3.xml 2025-12-04T11:39:12.5714027Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5669ef320dd02577.xml 2025-12-04T11:39:12.6013392Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c3e4151039d68fa8.xml 2025-12-04T11:39:12.6294589Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-024c7a32a248042e.xml 2025-12-04T11:39:12.6574545Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c97e92ee7b028ae3.xml 2025-12-04T11:39:12.6910288Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d8dd34d3359043f5.xml 2025-12-04T11:39:12.7190673Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-934a5e5071fe9c36.xml 2025-12-04T11:39:12.7496126Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d42c1866d87ab3db.xml 2025-12-04T11:39:12.7990961Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d46cf2452bc01536.xml 2025-12-04T11:39:12.8310680Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8f4f39fa980e82f2.xml 2025-12-04T11:39:12.8644918Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0652764081d6da21.xml 2025-12-04T11:39:12.8906342Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-610c219c10381302.xml 2025-12-04T11:39:12.9200027Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ff765bddb1f8ea71.xml 2025-12-04T11:39:12.9546031Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9be5a3fd2574b057.xml 2025-12-04T11:39:12.9845029Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-df5bf940ae0a6258.xml 2025-12-04T11:39:13.0133446Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a70e67bbc896acc2.xml 2025-12-04T11:39:13.0437241Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e5a6bcea490b9fda.xml 2025-12-04T11:39:13.0775605Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1d02ec45c02f60fe.xml 2025-12-04T11:39:13.1053722Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-589f53701a6c7b6a.xml 2025-12-04T11:39:13.1405149Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-37a88acccbf1ecd0.xml 2025-12-04T11:39:13.1750934Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9e0e61cd971b69bb.xml 2025-12-04T11:39:13.2064635Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a25ed88c721098f2.xml 2025-12-04T11:39:13.2327287Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c9d57215b8ca5d9f.xml 2025-12-04T11:39:13.2659472Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-07a2c7293983a21f.xml 2025-12-04T11:39:13.2977848Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e7d26bc200bae427.xml 2025-12-04T11:39:13.3264040Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c4303c90119175e3.xml 2025-12-04T11:39:13.3566840Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b08f37975720f6b4.xml 2025-12-04T11:39:13.3983808Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-75b604dc9c04c4c7.xml 2025-12-04T11:39:13.4264569Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8dc1d7aea93bdb34.xml 2025-12-04T11:39:13.4590820Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4a3867d6b6932f10.xml 2025-12-04T11:39:13.4903932Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0318829c01be08fa.xml 2025-12-04T11:39:13.5205385Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-693b8cadac47d9c2.xml 2025-12-04T11:39:13.5526089Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0ecc816e8885b9bc.xml 2025-12-04T11:39:13.5826470Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d3419121e211fb91.xml 2025-12-04T11:39:13.6138328Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2c51cc1e26d5acf3.xml 2025-12-04T11:39:13.6446117Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4746fedc7b470508.xml 2025-12-04T11:39:13.6745055Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d82b54717ce755b5.xml 2025-12-04T11:39:13.7014951Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-730ab8cf8f5003b5.xml 2025-12-04T11:39:13.7366900Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ef621cb706a80f36.xml 2025-12-04T11:39:13.8034862Z Running distributed/fsdp/test_fsdp_unshard_params 1/1 ... [2025-12-04 11:39:13.803272][9984.74090876] 2025-12-04T11:39:13.8035527Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:39:13.8037893Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_unshard_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:39:13.803603] 2025-12-04T11:40:33.2210698Z 2025-12-04T11:40:33.2214103Z distributed/fsdp/test_fsdp_unshard_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_62027c77e09bb2e8_.log 2025-12-04T11:40:33.2223273Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_named_parameters_and_buffers, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_param_data, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_recurse, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_respects_reshard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_singleton_param_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_submodule, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_core, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_none_grads, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_param_data_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_writeback_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_offload_to_cpu_no_shard_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_rank0_only_with_writeback_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_backward_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_forward_raises 2025-12-04T11:40:33.2232195Z 2025-12-04T11:40:33.2232642Z Finished distributed/fsdp/test_fsdp_unshard_params 1/1 ... [2025-12-04 11:40:33.220809][10064.158441338], took 1.32min 2025-12-04T11:40:33.2473326Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_unshard_params/distributed.fsdp.test_fsdp_unshard_params-49f6028e4feadb15.xml 2025-12-04T11:40:33.3758166Z Running distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2025-12-04 11:40:33.375589][10064.31322455] 2025-12-04T11:40:33.3758864Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:40:33.3761901Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:40:33.375916] 2025-12-04T11:41:05.7177916Z 2025-12-04T11:41:05.7179150Z distributed/_shard/sharded_tensor/ops/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_1f62894d62d48806_.log 2025-12-04T11:41:05.7182130Z Running 3 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_kaiming_uniform, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_normal, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_uniform 2025-12-04T11:41:05.7184234Z 2025-12-04T11:41:05.7184711Z Finished distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2025-12-04 11:41:05.717469][10096.655098893], took 0.54min 2025-12-04T11:41:05.7438897Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-b148a45a231b8de9.xml 2025-12-04T11:41:05.8743456Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 ... [2025-12-04 11:41:05.873802][10096.811438462] 2025-12-04T11:41:05.8744202Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:41:05.8745569Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:41:05.874150] 2025-12-04T11:41:22.7825985Z 2025-12-04T11:41:22.7827469Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_c34f896008a30836_.log 2025-12-04T11:41:22.7830024Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_rowwise 2025-12-04T11:41:22.7831516Z 2025-12-04T11:41:22.7832396Z Finished distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 ... [2025-12-04 11:41:22.782215][10113.719850728], took 0.28min 2025-12-04T11:41:22.8089963Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding_bag/distributed._shard.sharded_tensor.ops.test_embedding_bag-0fb42e6113dec695.xml 2025-12-04T11:41:22.9414716Z Running distributed/test_c10d_spawn_gloo 1/1 ... [2025-12-04 11:41:22.940888][10113.878525553] 2025-12-04T11:41:22.9415336Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:41:22.9416634Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:41:22.941235] 2025-12-04T11:42:49.6033092Z 2025-12-04T11:42:49.6034211Z distributed/test_c10d_spawn_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_gloo_1.1_0181e7af55ebaf0a_.log 2025-12-04T11:42:49.6040694Z Running 11 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2025-12-04T11:42:49.6046744Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu 2025-12-04T11:42:49.6047890Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda 2025-12-04T11:42:49.6049006Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn 2025-12-04T11:42:49.6050105Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather 2025-12-04T11:42:49.6051179Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all 2025-12-04T11:42:49.6052290Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single 2025-12-04T11:42:49.6053382Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce 2025-12-04T11:42:49.6054446Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast 2025-12-04T11:42:49.6055493Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather 2025-12-04T11:42:49.6056519Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce 2025-12-04T11:42:49.6057540Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2025-12-04T11:42:49.6058139Z 2025-12-04T11:42:49.6058510Z Finished distributed/test_c10d_spawn_gloo 1/1 ... [2025-12-04 11:42:49.603330][10200.540965413], took 1.44min 2025-12-04T11:42:49.6387210Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e3270986f520c455.xml 2025-12-04T11:42:49.7210492Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9f07ad58b8cd3f14.xml 2025-12-04T11:42:49.7568043Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-38ae8a6c9446176c.xml 2025-12-04T11:42:49.7864906Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9be7dc10fc456dd4.xml 2025-12-04T11:42:49.8190800Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a4bb8d2a63c3e77a.xml 2025-12-04T11:42:49.8522590Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a65cb5796fd83f18.xml 2025-12-04T11:42:49.8800388Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-44d720723f83b050.xml 2025-12-04T11:42:49.9149733Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a0db0c7d2253e897.xml 2025-12-04T11:42:49.9487075Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-4d41a04cacb8591f.xml 2025-12-04T11:42:49.9771510Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-79c50cf35aedde1f.xml 2025-12-04T11:42:50.0126560Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e36163ffaa730c12.xml 2025-12-04T11:42:50.0854380Z Running distributed/test_c10d_ucc 1/1 ... [2025-12-04 11:42:50.084781][10201.022418219] 2025-12-04T11:42:50.0854985Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:42:50.0856261Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:42:50.085125] 2025-12-04T11:42:51.8018593Z 2025-12-04T11:42:51.8019610Z distributed/test_c10d_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ucc_1.1_511a2327d2886668_.log 2025-12-04T11:42:51.8020457Z 2025-12-04T11:42:51.8701086Z Finished distributed/test_c10d_ucc 1/1 ... [2025-12-04 11:42:51.801286][10202.7389216], took 0.03min 2025-12-04T11:42:51.8701980Z Running distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2025-12-04 11:42:51.869472][10202.807110049] 2025-12-04T11:42:51.8702629Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:42:51.8703906Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:42:51.869841] 2025-12-04T11:49:42.0281479Z 2025-12-04T11:49:42.0282676Z distributed/fsdp/test_fsdp_use_orig_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_bd37660784d144d8_.log 2025-12-04T11:49:42.0300365Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_trainability, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_fsdp_compile, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_multiple_optimizers, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsParamAccess::test_access_params_after_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_grad_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_no_reshard_and_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_param_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_between_fwd_and_bwd_for_no_reshard_raises, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_shape_mismatch, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsFQNs::test_named_parameters_in_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_correctness, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsInit::test_non_uniform_requires_grad, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cpu, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cuda 2025-12-04T11:49:42.0317399Z 2025-12-04T11:49:42.0317854Z Finished distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2025-12-04 11:49:42.027981][10612.965616448], took 6.84min 2025-12-04T11:49:42.0566271Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_use_orig_params/distributed.fsdp.test_fsdp_use_orig_params-c1df23351bfa5ee8.xml 2025-12-04T11:49:42.6723237Z Uploading artifacts took 0.44 seconds 2025-12-04T11:49:42.6726021Z Running distributed/test_c10d_common 1/1 ... [2025-12-04 11:49:42.672439][10613.610075223] 2025-12-04T11:49:42.6726622Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:49:42.6729812Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:49:42.672788] 2025-12-04T11:53:17.2350755Z 2025-12-04T11:53:17.2352163Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_41bda74550716a01_.log 2025-12-04T11:53:17.2365649Z Running 27 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_abort, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_canonicalize_helper, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_shutdown, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2025-12-04T11:53:17.2378855Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2025-12-04T11:53:17.2379887Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2025-12-04T11:53:17.2381048Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2025-12-04T11:53:17.2382200Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2025-12-04T11:53:17.2383377Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2025-12-04T11:53:17.2384361Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2025-12-04T11:53:17.2385281Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_abort 2025-12-04T11:53:17.2386359Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2025-12-04T11:53:17.2387763Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2025-12-04T11:53:17.2389010Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_canonicalize_helper 2025-12-04T11:53:17.2390157Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2025-12-04T11:53:17.2391274Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2025-12-04T11:53:17.2392563Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2025-12-04T11:53:17.2393869Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2025-12-04T11:53:17.2395009Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2025-12-04T11:53:17.2396083Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_shutdown 2025-12-04T11:53:17.2397350Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_default_process_group 2025-12-04T11:53:17.2398785Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2025-12-04T11:53:17.2400382Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2025-12-04T11:53:17.2401650Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2025-12-04T11:53:17.2402616Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2025-12-04T11:53:17.2403532Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2025-12-04T11:53:17.2404431Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2025-12-04T11:53:17.2405333Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank 2025-12-04T11:53:17.2406305Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback 2025-12-04T11:53:17.2407256Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv 2025-12-04T11:53:17.2408175Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2025-12-04T11:53:17.2408733Z 2025-12-04T11:53:17.2409087Z Finished distributed/test_c10d_common 1/1 ... [2025-12-04 11:53:17.235206][10828.172839505], took 3.58min 2025-12-04T11:53:17.2634664Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-093e5ad7fc260f19.xml 2025-12-04T11:53:17.3605266Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-60dcb39f38cfdee4.xml 2025-12-04T11:53:17.3883475Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-c9c5102c76f0a9a2.xml 2025-12-04T11:53:17.4184292Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-312bbf42e62ed9b6.xml 2025-12-04T11:53:17.4470070Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-75c9bd860618416d.xml 2025-12-04T11:53:17.4780378Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-104493b7b71c2fe7.xml 2025-12-04T11:53:17.5083604Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-d6965728c111189f.xml 2025-12-04T11:53:17.5389610Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-550261c769de2900.xml 2025-12-04T11:53:17.5691567Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-c61f939b9c1212d0.xml 2025-12-04T11:53:17.6043574Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-f26443fa9fd7d2be.xml 2025-12-04T11:53:17.6320690Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-33f9aeac8ff2d88d.xml 2025-12-04T11:53:17.6652762Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-6f219815bc6782ef.xml 2025-12-04T11:53:17.6946610Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-1f65ab45ec69dc91.xml 2025-12-04T11:53:17.7227281Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a7ef02fe48ea3a40.xml 2025-12-04T11:53:17.7521823Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a8622e73769262d0.xml 2025-12-04T11:53:17.7844578Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-3ba3606e730f6b82.xml 2025-12-04T11:53:17.8227112Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-48cf7919df9353bf.xml 2025-12-04T11:53:17.8541220Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-5927afc313986450.xml 2025-12-04T11:53:17.8863599Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-e564db12bfa2b80c.xml 2025-12-04T11:53:17.9261172Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-17d2371d605f4c7c.xml 2025-12-04T11:53:17.9653041Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b0038959c806a90a.xml 2025-12-04T11:53:17.9989331Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-5cc7624c76b4d82c.xml 2025-12-04T11:53:18.0329088Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-ef399536ef6f172f.xml 2025-12-04T11:53:18.0680775Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b77e93b7526c6e31.xml 2025-12-04T11:53:18.0999751Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-52e140e51a31b98a.xml 2025-12-04T11:53:18.1341390Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b7bc54ed87c9274a.xml 2025-12-04T11:53:18.1682915Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-927eadde7027cdb3.xml 2025-12-04T11:53:18.2397302Z Running distributed/fsdp/test_fsdp_mixed_precision 1/1 ... [2025-12-04 11:53:18.239494][10829.17713037] 2025-12-04T11:53:18.2397983Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:53:18.2400092Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:53:18.239822] 2025-12-04T11:59:03.2889338Z 2025-12-04T11:59:03.2890508Z distributed/fsdp/test_fsdp_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_79ffb8eaa6b21953_.log 2025-12-04T11:59:03.2942261Z Running 66 items in this shard: test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_buffer_dtype_no_root_handle, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_eval_root_cast_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_buffers, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_comm, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_input_grads_with_param_mixed_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_resnet, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_False, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_True, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_default, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_only_params_and_bufs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_params_and_reduce_diff, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_reduce, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_e2e_full_shard, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionIgnoredModules::test_mixed_precision_with_ignored_module, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_external_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPTrainEval::test_train_ema_eval_flow 2025-12-04T11:59:03.2991768Z 2025-12-04T11:59:03.2992227Z Finished distributed/fsdp/test_fsdp_mixed_precision 1/1 ... [2025-12-04 11:59:03.288735][11174.226369114], took 5.75min 2025-12-04T11:59:03.3207064Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_fsdp_mixed_precision/distributed.fsdp.test_fsdp_mixed_precision-c50711adaece2e48.xml 2025-12-04T11:59:03.4589696Z Running distributed/test_c10d_nccl 2/3 ... [2025-12-04 11:59:03.458401][11174.396037773] 2025-12-04T11:59:03.4590289Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2025-12-04T11:59:03.4591612Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2025-12-04 11:59:03.458760] 2025-12-04T12:13:08.4897393Z 2025-12-04T12:13:08.4901007Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_3f865a26beb29e8c_.log 2025-12-04T12:13:08.4944127Z Running 83 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_scalable_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_cuda_event_cache_mthd_race, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_deterministic_mode_no_break, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_basic, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_multiple_iterations, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_window_registration, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_unwaited, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_multiple_resets_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_circular_buffer_full_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_wraparound_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_wraparound_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2025-12-04T12:13:08.4984287Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2025-12-04T12:13:08.4985344Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_scalable_init 2025-12-04T12:13:08.4986396Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg 2025-12-04T12:13:08.4987485Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup 2025-12-04T12:13:08.4988781Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_cuda_event_cache_mthd_race 2025-12-04T12:13:08.4989962Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg 2025-12-04T12:13:08.4991146Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_deterministic_mode_no_break 2025-12-04T12:13:08.4992346Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0 2025-12-04T12:13:08.4993517Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16 2025-12-04T12:13:08.4994578Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64 2025-12-04T12:13:08.4995676Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn 2025-12-04T12:13:08.4996770Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check 2025-12-04T12:13:08.4997776Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter 2025-12-04T12:13:08.4998983Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False 2025-12-04T12:13:08.5000137Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p 2025-12-04T12:13:08.5001134Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0 2025-12-04T12:13:08.5002141Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc 2025-12-04T12:13:08.5003109Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_basic 2025-12-04T12:13:08.5004136Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_shrink_group_multiple_iterations 2025-12-04T12:13:08.5005211Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True 2025-12-04T12:13:08.5006341Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2025-12-04T12:13:08.5007512Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2025-12-04T12:13:08.5008591Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2025-12-04T12:13:08.5009704Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2025-12-04T12:13:08.5010844Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2025-12-04T12:13:08.5012013Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2025-12-04T12:13:08.5013259Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2025-12-04T12:13:08.5014585Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2025-12-04T12:13:08.5015800Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2025-12-04T12:13:08.5016995Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2025-12-04T12:13:08.5018124Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2025-12-04T12:13:08.5019159Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2025-12-04T12:13:08.5020163Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2025-12-04T12:13:08.5021295Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2025-12-04T12:13:08.5022470Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2025-12-04T12:13:08.5023503Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2025-12-04T12:13:08.5025141Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2025-12-04T12:13:08.5026565Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2025-12-04T12:13:08.5028080Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2025-12-04T12:13:08.5029516Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2025-12-04T12:13:08.5030753Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2025-12-04T12:13:08.5031943Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2025-12-04T12:13:08.5033134Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2025-12-04T12:13:08.5034186Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2025-12-04T12:13:08.5044078Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_window_registration 2025-12-04T12:13:08.5045191Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl 2025-12-04T12:13:08.5046194Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2025-12-04T12:13:08.5047137Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2025-12-04T12:13:08.5048017Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2025-12-04T12:13:08.5048878Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2025-12-04T12:13:08.5049833Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2025-12-04T12:13:08.5050775Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2025-12-04T12:13:08.5051752Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2025-12-04T12:13:08.5052716Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k 2025-12-04T12:13:08.5053738Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_unwaited 2025-12-04T12:13:08.5054769Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2025-12-04T12:13:08.5056186Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_default_process_group 2025-12-04T12:13:08.5057473Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False 2025-12-04T12:13:08.5058592Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True 2025-12-04T12:13:08.5059683Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False 2025-12-04T12:13:08.5060721Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True 2025-12-04T12:13:08.5061711Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False 2025-12-04T12:13:08.5062703Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False 2025-12-04T12:13:08.5063620Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2025-12-04T12:13:08.5064621Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True 2025-12-04T12:13:08.5065727Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True 2025-12-04T12:13:08.5066896Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True 2025-12-04T12:13:08.5068392Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True 2025-12-04T12:13:08.5069688Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2025-12-04T12:13:08.5071029Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2025-12-04T12:13:08.5072378Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2025-12-04T12:13:08.5073658Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_multiple_resets_timing_enabled_True 2025-12-04T12:13:08.5074909Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_circular_buffer_full_timing_enabled_True 2025-12-04T12:13:08.5076128Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_timing_enabled_False 2025-12-04T12:13:08.5077287Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_wraparound_timing_enabled_False 2025-12-04T12:13:08.5078493Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_fr_record_reset_wraparound_timing_enabled_True 2025-12-04T12:13:08.5079821Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2025-12-04T12:13:08.5081075Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2025-12-04T12:13:08.5082208Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True 2025-12-04T12:13:08.5083436Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False 2025-12-04T12:13:08.5084592Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True 2025-12-04T12:13:08.5085753Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False 2025-12-04T12:13:08.5086890Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2025-12-04T12:13:08.5087499Z 2025-12-04T12:13:08.5087832Z Finished distributed/test_c10d_nccl 2/3 ... [2025-12-04 12:13:08.490790][12019.428405595], took 14.08min 2025-12-04T12:13:08.5226924Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-81f85fc9fa3db30b.xml 2025-12-04T12:13:08.6080923Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0df1dc2a08194e48.xml 2025-12-04T12:13:08.6497973Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-43498c0a89946df9.xml 2025-12-04T12:13:08.6816545Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-af4c89e072116764.xml 2025-12-04T12:13:08.7124400Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-98f78a29e90359ed.xml 2025-12-04T12:13:08.7413645Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-30adfd6c7b8ecf71.xml 2025-12-04T12:13:08.7687489Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-581dd59091205273.xml 2025-12-04T12:13:08.8029866Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d528c59f12482b1.xml 2025-12-04T12:13:08.8340951Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d6315101734e178a.xml 2025-12-04T12:13:08.8664418Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d89c8d6fc7b73a1b.xml 2025-12-04T12:13:08.8993642Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c23cf3f46128ac0f.xml 2025-12-04T12:13:08.9296885Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3a09c39260c6bdcb.xml 2025-12-04T12:13:08.9617812Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f486758b2548beb0.xml 2025-12-04T12:13:08.9944204Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-36d76263f3b24111.xml 2025-12-04T12:13:09.0264559Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f1c94044951be399.xml 2025-12-04T12:13:09.0557709Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b9df70724e6dcd35.xml 2025-12-04T12:13:09.0873604Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fbd78536cd21cf97.xml 2025-12-04T12:13:09.1167594Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ca3fc647503e2182.xml 2025-12-04T12:13:09.1483650Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-15e1be913d8837e9.xml 2025-12-04T12:13:09.1783612Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8a7b7a30260424a5.xml 2025-12-04T12:13:09.2071754Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-13f85e6fb9609878.xml 2025-12-04T12:13:09.2353754Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d09ed72d675bfdc8.xml 2025-12-04T12:13:09.2712379Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3eb78e74581a557b.xml 2025-12-04T12:13:09.3096786Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-90acb82834a56757.xml 2025-12-04T12:13:09.3365160Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a1738cfdeff44f2e.xml 2025-12-04T12:13:09.3667707Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7936e4b0274d7ad6.xml 2025-12-04T12:13:09.3957467Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bd821bd95ef10bd6.xml 2025-12-04T12:13:09.4255817Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5839fc6b3294dc3b.xml 2025-12-04T12:13:09.4566572Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-be22aa9f1a9dc572.xml 2025-12-04T12:13:09.4864073Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-63d98e3d16f90c98.xml 2025-12-04T12:13:09.5175868Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-93df4bdbf125060c.xml 2025-12-04T12:13:09.5446993Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c1c3a483347ae16e.xml 2025-12-04T12:13:09.5775369Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5c6d35f4b8178e66.xml 2025-12-04T12:13:09.6095236Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f4a815301947af44.xml 2025-12-04T12:13:09.6443554Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ea05d17fa83aaab3.xml 2025-12-04T12:13:09.6766226Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c99c7919907a8f51.xml 2025-12-04T12:13:09.7068084Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3228075a6adb947.xml 2025-12-04T12:13:09.7375453Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5f3dfac751b51f15.xml 2025-12-04T12:13:09.7683747Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9836e36a022fec7d.xml 2025-12-04T12:13:09.7950350Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-481f5a9e007c92bc.xml 2025-12-04T12:13:09.8254015Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8e5333f0f8c7af12.xml 2025-12-04T12:13:09.8565210Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-27c65021619c5afb.xml 2025-12-04T12:13:09.8913532Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3bc159f1fc89dbaa.xml 2025-12-04T12:13:09.9426127Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4183a04f60a93385.xml 2025-12-04T12:13:09.9750723Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b6ee0414e522cc1e.xml 2025-12-04T12:13:10.0043360Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ba060feb7f6eb8f7.xml 2025-12-04T12:13:10.0371888Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3aa486503236fa6.xml 2025-12-04T12:13:10.0683751Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a218cf86ddf03214.xml 2025-12-04T12:13:10.1014931Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fb621eae387ef1df.xml 2025-12-04T12:13:10.1310043Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-80e6fc2adcb64c0a.xml 2025-12-04T12:13:10.1590745Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2323c2353e236ae2.xml 2025-12-04T12:13:10.1892808Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-21c4771a3c9fd2ac.xml 2025-12-04T12:13:10.2167450Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e704bc72dcc2db74.xml 2025-12-04T12:13:10.2551804Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8da80ebf3429571e.xml 2025-12-04T12:13:10.2866068Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-aad331620921f562.xml 2025-12-04T12:13:10.3163273Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4d876218f2c25b17.xml 2025-12-04T12:13:10.3445508Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8a4e259673457968.xml 2025-12-04T12:13:10.3751613Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d2017d482dc7eebb.xml 2025-12-04T12:13:10.4126752Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cd8336cfdb34c878.xml 2025-12-04T12:13:10.4477919Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-be2011792fadd5cd.xml 2025-12-04T12:13:10.4869813Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fd2ba77f6fee2f8a.xml 2025-12-04T12:13:10.5190560Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-da6ff71bbbbcfd5b.xml 2025-12-04T12:13:10.5496098Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-00c7457a094406e2.xml 2025-12-04T12:13:10.5857219Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f38ba7e725b3376c.xml 2025-12-04T12:13:10.6405968Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a85d18d4001a0d1f.xml 2025-12-04T12:13:10.6706558Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-77e675aec14c1ef5.xml 2025-12-04T12:13:10.7026102Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-679877f4d338af19.xml 2025-12-04T12:13:10.7365300Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8d26022f0d5f0e5b.xml 2025-12-04T12:13:10.8031798Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d52baf36617de6c.xml 2025-12-04T12:13:10.8455332Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-97cfc974aeae8bab.xml 2025-12-04T12:13:10.8747434Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dc857ca5b8949f50.xml 2025-12-04T12:13:10.9065108Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-68d634b03e6f1cf2.xml 2025-12-04T12:13:10.9425819Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-797d0b27326bd9f0.xml 2025-12-04T12:13:10.9912147Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7f4f2591d711a574.xml 2025-12-04T12:13:11.0174894Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-54ee371b40f31a9b.xml 2025-12-04T12:13:11.0495417Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bc7d0e9f12eb7a6c.xml 2025-12-04T12:13:11.0845272Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4b8aea19c365479d.xml 2025-12-04T12:13:11.1163630Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ace037ae618597c7.xml 2025-12-04T12:13:11.1526760Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8980b265f0dcd572.xml 2025-12-04T12:13:11.1871338Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2f6c447cb3d8d9ba.xml 2025-12-04T12:13:11.2176107Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-902c7c5c4fac1c79.xml 2025-12-04T12:13:11.2457433Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3399e4ec5bb86d0f.xml 2025-12-04T12:13:11.2736257Z Parsing testcases for test report: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ee7e5984f1b0da3e.xml 2025-12-04T12:13:11.7790721Z Uploading artifacts took 0.43 seconds 2025-12-04T12:13:15.9034073Z Running test batch 'tests to run' cost 11205.92 seconds 2025-12-04T12:13:15.9041906Z Emitting td_test_failure_stats_v2 2025-12-04T12:13:15.9045463Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850395_9cccfef6d10a11f09d020242ac110002 2025-12-04T12:13:16.0364101Z Done! Finish writing document to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850395_9cccfef6d10a11f09d020242ac110002 2025-12-04T12:13:16.0366545Z Emitting td_test_failure_stats_v2 2025-12-04T12:13:16.0368611Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850396_9ce13056d10a11f09d020242ac110002 2025-12-04T12:13:16.0701023Z Done! Finish writing document to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850396_9ce13056d10a11f09d020242ac110002 2025-12-04T12:13:16.0703698Z Emitting td_test_failure_stats_v2 2025-12-04T12:13:16.0705787Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850396_9ce6550ed10a11f09d020242ac110002 2025-12-04T12:13:16.1037314Z Done! Finish writing document to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850396_9ce6550ed10a11f09d020242ac110002 2025-12-04T12:13:16.1038490Z Emitting td_test_failure_stats_v2 2025-12-04T12:13:16.1040373Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850396_9ceb70d4d10a11f09d020242ac110002 2025-12-04T12:13:16.1342037Z Done! Finish writing document to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850396_9ceb70d4d10a11f09d020242ac110002 2025-12-04T12:13:16.1344319Z Emitting td_test_failure_stats_v2 2025-12-04T12:13:16.1345501Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850396_9cf0185ad10a11f09d020242ac110002 2025-12-04T12:13:16.1671000Z Done! Finish writing document to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1764850396_9cf0185ad10a11f09d020242ac110002 2025-12-04T12:13:16.1671949Z distributed/fsdp/test_fsdp_uneven 1/1 failed! 2025-12-04T12:13:16.1672391Z distributed/fsdp/test_fsdp_input 1/1 failed! 2025-12-04T12:13:16.1672831Z distributed/fsdp/test_fsdp_traversal 1/1 failed! 2025-12-04T12:13:16.1673280Z distributed/fsdp/test_fsdp_checkpoint 1/1 failed! 2025-12-04T12:13:16.1673818Z distributed/fsdp/test_fsdp_comm 1/1 failed! 2025-12-04T12:13:16.9950550Z 2025-12-04T12:13:16.9950955Z real 186m52.693s 2025-12-04T12:13:16.9951311Z user 487m23.484s 2025-12-04T12:13:16.9951561Z sys 232m20.474s 2025-12-04T12:13:16.9951824Z + sccache_epilogue 2025-12-04T12:13:16.9952157Z + echo '::group::Sccache Compilation Log' 2025-12-04T12:13:16.9952851Z ##[group]Sccache Compilation Log 2025-12-04T12:13:16.9953256Z + echo '=================== sccache compilation log ===================' 2025-12-04T12:13:16.9953741Z =================== sccache compilation log =================== 2025-12-04T12:13:16.9954472Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-12-04T12:13:17.0093343Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-12-04T12:13:17.0094139Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-12-04T12:13:17.0094695Z + sccache --show-stats 2025-12-04T12:13:17.0114030Z Compile requests 957 2025-12-04T12:13:17.0114447Z Compile requests executed 0 2025-12-04T12:13:17.0114806Z Cache hits 0 2025-12-04T12:13:17.0115155Z Cache misses 0 2025-12-04T12:13:17.0115512Z Cache hits rate - 2025-12-04T12:13:17.0115873Z Cache timeouts 0 2025-12-04T12:13:17.0116223Z Cache read errors 0 2025-12-04T12:13:17.0116633Z Forced recaches 0 2025-12-04T12:13:17.0116999Z Cache write errors 0 2025-12-04T12:13:17.0117454Z Cache errors 0 2025-12-04T12:13:17.0117816Z Compilations 0 2025-12-04T12:13:17.0118169Z Compilation failures 0 2025-12-04T12:13:17.0118545Z Non-cacheable compilations 0 2025-12-04T12:13:17.0118918Z Non-cacheable calls 2 2025-12-04T12:13:17.0119272Z Non-compilation calls 955 2025-12-04T12:13:17.0119651Z Unsupported compiler calls 0 2025-12-04T12:13:17.0120214Z Average cache write 0.000 s 2025-12-04T12:13:17.0120571Z Average compiler 0.000 s 2025-12-04T12:13:17.0120939Z Average cache read hit 0.000 s 2025-12-04T12:13:17.0121320Z Failed distributed compilations 0 2025-12-04T12:13:17.0121566Z 2025-12-04T12:13:17.0121690Z Non-cacheable reasons: 2025-12-04T12:13:17.0121970Z -E 2 2025-12-04T12:13:17.0122213Z 2025-12-04T12:13:17.0122472Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T12:13:17.0122994Z Version (client) 0.10.0 2025-12-04T12:13:17.0123333Z + sccache --stop-server 2025-12-04T12:13:17.0141693Z Stopping sccache server... 2025-12-04T12:13:17.0142074Z Compile requests 957 2025-12-04T12:13:17.0142446Z Compile requests executed 0 2025-12-04T12:13:17.0142787Z Cache hits 0 2025-12-04T12:13:17.0143178Z Cache misses 0 2025-12-04T12:13:17.0143537Z Cache hits rate - 2025-12-04T12:13:17.0143892Z Cache timeouts 0 2025-12-04T12:13:17.0144350Z Cache read errors 0 2025-12-04T12:13:17.0144816Z Forced recaches 0 2025-12-04T12:13:17.0145147Z Cache write errors 0 2025-12-04T12:13:17.0145486Z Cache errors 0 2025-12-04T12:13:17.0145833Z Compilations 0 2025-12-04T12:13:17.0146173Z Compilation failures 0 2025-12-04T12:13:17.0146548Z Non-cacheable compilations 0 2025-12-04T12:13:17.0146910Z Non-cacheable calls 2 2025-12-04T12:13:17.0147405Z Non-compilation calls 955 2025-12-04T12:13:17.0147957Z Unsupported compiler calls 0 2025-12-04T12:13:17.0148333Z Average cache write 0.000 s 2025-12-04T12:13:17.0148711Z Average compiler 0.000 s 2025-12-04T12:13:17.0149071Z Average cache read hit 0.000 s 2025-12-04T12:13:17.0149457Z Failed distributed compilations 0 2025-12-04T12:13:17.0149706Z 2025-12-04T12:13:17.0149830Z Non-cacheable reasons: 2025-12-04T12:13:17.0150112Z -E 2 2025-12-04T12:13:17.0150356Z 2025-12-04T12:13:17.0150622Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-12-04T12:13:17.0151148Z Version (client) 0.10.0 2025-12-04T12:13:17.0151506Z + echo ::endgroup:: 2025-12-04T12:13:17.0152051Z ##[endgroup] 2025-12-04T12:13:17.0152386Z + cleanup_workspace 2025-12-04T12:13:17.0153224Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-12-04T12:13:17.0154447Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-12-04T12:13:17.0155176Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-12-04T12:13:17.0155720Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-12-04T12:13:17.0156469Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-12-04T12:13:17.0157519Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-12-04T12:13:17.0158184Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-12-04T12:13:17.6832840Z ##[error]Process completed with exit code 1. 2025-12-04T12:13:17.6904869Z Prepare all required actions 2025-12-04T12:13:17.6905302Z Getting action download info 2025-12-04T12:13:17.9077141Z ##[group]Run ./.github/actions/pytest-cache-upload 2025-12-04T12:13:17.9077621Z with: 2025-12-04T12:13:17.9077876Z cache_dir: .pytest_cache 2025-12-04T12:13:17.9078176Z shard: 2 2025-12-04T12:13:17.9078453Z sha: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T12:13:17.9078848Z test_config: distributed 2025-12-04T12:13:17.9079239Z job_identifier: trunk_linux-jammy-cuda12.8-py3.10-gcc11 2025-12-04T12:13:17.9079660Z env: 2025-12-04T12:13:17.9080013Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:17.9080357Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:17.9080687Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:17.9081248Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:17.9081758Z ##[endgroup] 2025-12-04T12:13:17.9116608Z ##[group]Run nick-fields/retry@v3.0.0 2025-12-04T12:13:17.9117017Z with: 2025-12-04T12:13:17.9117238Z shell: bash 2025-12-04T12:13:17.9117497Z timeout_minutes: 5 2025-12-04T12:13:17.9117779Z max_attempts: 5 2025-12-04T12:13:17.9118049Z retry_wait_seconds: 30 2025-12-04T12:13:17.9118435Z command: set -eu python3 -m pip install boto3==1.35.42 2025-12-04T12:13:17.9118874Z polling_interval_seconds: 1 2025-12-04T12:13:17.9119202Z warning_on_retry: true 2025-12-04T12:13:17.9119493Z continue_on_error: false 2025-12-04T12:13:17.9119781Z env: 2025-12-04T12:13:17.9120135Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:17.9120392Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:17.9120712Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:17.9121289Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:17.9121789Z ##[endgroup] 2025-12-04T12:13:18.2663976Z Defaulting to user installation because normal site-packages is not writeable 2025-12-04T12:13:19.4771244Z Collecting boto3==1.35.42 2025-12-04T12:13:19.4955062Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-12-04T12:13:20.8477710Z Collecting botocore<1.36.0,>=1.35.42 2025-12-04T12:13:20.8522381Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-12-04T12:13:21.0216549Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-12-04T12:13:21.0652627Z Collecting s3transfer<0.11.0,>=0.10.0 2025-12-04T12:13:21.0689169Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-12-04T12:13:21.0789471Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (2.8.1) 2025-12-04T12:13:21.0799347Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-12-04T12:13:21.3325927Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.15.0) 2025-12-04T12:13:21.4287085Z Installing collected packages: botocore, s3transfer, boto3 2025-12-04T12:13:22.0105025Z Successfully installed boto3-1.35.42 botocore-1.35.99 s3transfer-0.10.4 2025-12-04T12:13:23.0004322Z Command completed after 1 attempt(s). 2025-12-04T12:13:23.0059838Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2025-12-04T12:13:23.0060330Z python3 .github/scripts/pytest_cache.py \ 2025-12-04T12:13:23.0060712Z  --upload \ 2025-12-04T12:13:23.0061057Z  --cache_dir "$GITHUB_WORKSPACE/$CACHE_DIR" \ 2025-12-04T12:13:23.0061479Z  --pr_identifier "$GITHUB_REF" \ 2025-12-04T12:13:23.0061864Z  --job_identifier "$JOB_IDENTIFIER" \ 2025-12-04T12:13:23.0062228Z  --sha "$SHA" \ 2025-12-04T12:13:23.0062541Z  --test_config "$TEST_CONFIG" \ 2025-12-04T12:13:23.0062892Z  --shard "$SHARD" \ 2025-12-04T12:13:23.0063184Z  --repo "$REPO" \ 2025-12-04T12:13:23.0063664Z  --temp_dir "$RUNNER_TEMP" \ 2025-12-04T12:13:23.0074112Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:23.0074639Z env: 2025-12-04T12:13:23.0074895Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:23.0075207Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:23.0075563Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:23.0076217Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:23.0076809Z CACHE_DIR: .pytest_cache 2025-12-04T12:13:23.0077206Z JOB_IDENTIFIER: trunk_linux-jammy-cuda12.8-py3.10-gcc11 2025-12-04T12:13:23.0077740Z SHA: ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32 2025-12-04T12:13:23.0078143Z TEST_CONFIG: distributed 2025-12-04T12:13:23.0078441Z SHARD: 2 2025-12-04T12:13:23.0078684Z REPO: pytorch/pytorch 2025-12-04T12:13:23.0078973Z ##[endgroup] 2025-12-04T12:13:23.3933260Z PR identifier for `refs/heads/main` is `96e092540d6b3c4076e3d2bc6f1f9013` 2025-12-04T12:13:23.3935726Z Uploading cache with args Namespace(upload=True, download=False, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', pr_identifier='refs/heads/main', job_identifier='trunk_linux-jammy-cuda12.8-py3.10-gcc11', sha='ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32', test_config='distributed', shard='2', repo='pytorch/pytorch', temp_dir='/home/ec2-user/actions-runner/_work/_temp', bucket=None) 2025-12-04T12:13:23.3937980Z Zipping /home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache 2025-12-04T12:13:23.3939517Z to /home/ec2-user/actions-runner/_work/_temp/zip-upload/pytest_cache/pytorch/pytorch/96e092540d6b3c4076e3d2bc6f1f9013/trunk_linux-jammy-cuda12_8-py3_10-gcc11/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32/distributed/2 2025-12-04T12:13:23.3941700Z Uploading /home/ec2-user/actions-runner/_work/_temp/zip-upload/pytest_cache/pytorch/pytorch/96e092540d6b3c4076e3d2bc6f1f9013/trunk_linux-jammy-cuda12_8-py3_10-gcc11/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32/distributed/2.zip 2025-12-04T12:13:23.3943599Z to s3://gha-artifacts/pytest_cache/pytorch/pytorch/96e092540d6b3c4076e3d2bc6f1f9013/trunk_linux-jammy-cuda12_8-py3_10-gcc11/ffd9b0fb4355e97af82fc42cf185c3ffa0fc0a32/distributed/2.zip 2025-12-04T12:13:23.4434375Z ##[group]Run cat test/**/*_toprint.log || true 2025-12-04T12:13:23.4434835Z cat test/**/*_toprint.log || true 2025-12-04T12:13:23.4441457Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:23.4441839Z env: 2025-12-04T12:13:23.4442111Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:23.4442385Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:23.4442697Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:23.4443284Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:23.4443795Z ##[endgroup] 2025-12-04T12:13:23.4539273Z cat: 'test/**/*_toprint.log': No such file or directory 2025-12-04T12:13:23.4581633Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-12-04T12:13:23.4582039Z kill "$MONITOR_SCRIPT_PID" 2025-12-04T12:13:23.4587909Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:23.4588352Z env: 2025-12-04T12:13:23.4588648Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:23.4588964Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:23.4589330Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:23.4589965Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:23.4590566Z MONITOR_SCRIPT_PID: 62785 2025-12-04T12:13:23.4590874Z ##[endgroup] 2025-12-04T12:13:23.4615630Z /home/ec2-user/actions-runner/_work/_temp/9e8ef872-9a35-45f3-b4d1-17bb2902bfae.sh: line 1: kill: (62785) - No such process 2025-12-04T12:13:23.4617872Z ##[error]Process completed with exit code 1. 2025-12-04T12:13:23.4752128Z Prepare all required actions 2025-12-04T12:13:23.4752615Z Getting action download info 2025-12-04T12:13:23.6100684Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-12-04T12:13:23.8435922Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-12-04T12:13:24.2564201Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-12-04T12:13:24.2564599Z with: 2025-12-04T12:13:24.2565050Z file-suffix: test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912 2025-12-04T12:13:24.2565605Z s3-bucket: gha-artifacts 2025-12-04T12:13:24.2565879Z env: 2025-12-04T12:13:24.2566115Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:24.2566406Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:24.2566744Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:24.2567461Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:24.2568047Z ##[endgroup] 2025-12-04T12:13:24.2593604Z ##[group]Run # Remove any previous test jsons if they exist 2025-12-04T12:13:24.2594147Z # Remove any previous test jsons if they exist 2025-12-04T12:13:24.2594573Z rm -f test-jsons-*.zip 2025-12-04T12:13:24.2595075Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-12-04T12:13:24.2601481Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:24.2602009Z env: 2025-12-04T12:13:24.2602228Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:24.2602511Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:24.2602842Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:24.2603407Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:24.2604120Z FILE_SUFFIX: test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912 2025-12-04T12:13:24.2604627Z ##[endgroup] 2025-12-04T12:13:24.2843010Z adding: test/test-reports/td_exclusions-2278c2ce744dc2b9295d.json (deflated 86%) 2025-12-04T12:13:24.2845662Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-98b5e89a175f7fbc.json (deflated 88%) 2025-12-04T12:13:24.2847247Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-bb030510459950ee.json (deflated 79%) 2025-12-04T12:13:24.2848654Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-0f19060d211e471b.json (deflated 91%) 2025-12-04T12:13:24.2850048Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-4e3b8ac35e315f80.json (deflated 79%) 2025-12-04T12:13:24.2851430Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-7270531c264bcf9a.json (stored 0%) 2025-12-04T12:13:24.2852831Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-ed7a8589598da4ef.json (deflated 79%) 2025-12-04T12:13:24.2854193Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-68059153dd3ae49c.json (deflated 79%) 2025-12-04T12:13:24.2855575Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-dd8d334d3533ae96.json (deflated 79%) 2025-12-04T12:13:24.2856958Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-571ae876e059f56e.json (deflated 79%) 2025-12-04T12:13:24.2858327Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-6f9ee7a291766819.json (deflated 79%) 2025-12-04T12:13:24.2859686Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-7369439069792ace.json (deflated 79%) 2025-12-04T12:13:24.2861281Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-f5da965ca571529c.json (stored 0%) 2025-12-04T12:13:24.2862702Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-551e66d027b33cdd.json (deflated 79%) 2025-12-04T12:13:24.2864159Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-8583a5ccabf54360.json (deflated 79%) 2025-12-04T12:13:24.2865675Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-58c9173ffcd43d6d.json (deflated 79%) 2025-12-04T12:13:24.2867117Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-3a68936c644e7b41.json (stored 0%) 2025-12-04T12:13:24.2868915Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-88a9f26e5fb62840.json (deflated 89%) 2025-12-04T12:13:24.2870528Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-52aebc2b2ad7446f.json (deflated 80%) 2025-12-04T12:13:24.2872063Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-c4e36de52e6d69c2.json (deflated 80%) 2025-12-04T12:13:24.2873580Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-448a44a24c52f556.json (stored 0%) 2025-12-04T12:13:24.2875037Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a25290001c1b014e.json (deflated 80%) 2025-12-04T12:13:24.2876438Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-cc80cfecc5abaf73.json (deflated 80%) 2025-12-04T12:13:24.2877853Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-00aad17389ddcea3.json (deflated 80%) 2025-12-04T12:13:24.2879241Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc7285fa7f4227.json (deflated 80%) 2025-12-04T12:13:24.2880715Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-69d9e5e2bc4a3a7d.json (deflated 88%) 2025-12-04T12:13:24.2882069Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c1c17128b5541633.json (deflated 80%) 2025-12-04T12:13:24.2883415Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-156a51594e715e30.json (deflated 80%) 2025-12-04T12:13:24.2884763Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2e6a295727102ba7.json (deflated 80%) 2025-12-04T12:13:24.2886114Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2bb3299d57ce6c2b.json (deflated 80%) 2025-12-04T12:13:24.2887467Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-b434b54a558efa89.json (deflated 80%) 2025-12-04T12:13:24.2888836Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-bd75c8cd2cf8ff9e.json (deflated 80%) 2025-12-04T12:13:24.2890202Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f7fa0727268f1a56.json (deflated 80%) 2025-12-04T12:13:24.2891557Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f90badc543da9251.json (deflated 88%) 2025-12-04T12:13:24.2892897Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-31be29224184e218.json (deflated 80%) 2025-12-04T12:13:24.2894341Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1c567b53e42c8343.json (deflated 80%) 2025-12-04T12:13:24.2895695Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-3db6e54b58890527.json (deflated 80%) 2025-12-04T12:13:24.2897046Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-5f8a5af0d826239e.json (deflated 91%) 2025-12-04T12:13:24.2898417Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1a3a2c29e874821d.json (deflated 88%) 2025-12-04T12:13:24.2899768Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-097bd188165cd189.json (deflated 91%) 2025-12-04T12:13:24.2901116Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-efc9fa1f0bd357d6.json (deflated 80%) 2025-12-04T12:13:24.2902508Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-648e0de30361d669.json (deflated 80%) 2025-12-04T12:13:24.2903848Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-21aecc6f8d181c25.json (deflated 80%) 2025-12-04T12:13:24.2905205Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-ab70b17c1ec5b8c5.json (deflated 88%) 2025-12-04T12:13:24.2906566Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e033904d3fffd97a.json (deflated 80%) 2025-12-04T12:13:24.2908169Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f030d3020f190621.json (deflated 79%) 2025-12-04T12:13:24.2909557Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-da8aed1c14f5aeeb.json (deflated 79%) 2025-12-04T12:13:24.2910957Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c91f300ede57a5d9.json (deflated 79%) 2025-12-04T12:13:24.2912359Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a2a9be50a03cddfc.json (deflated 79%) 2025-12-04T12:13:24.2913761Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-be7435212e20eea8.json (deflated 79%) 2025-12-04T12:13:24.2915163Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc23628278ddca.json (deflated 79%) 2025-12-04T12:13:24.2916536Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e7d51fdd1a2eae54.json (stored 0%) 2025-12-04T12:13:24.2918091Z adding: test/test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-19176bf0e30ddda9.json (deflated 78%) 2025-12-04T12:13:24.2919741Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-a72c588f9917f825.json (deflated 92%) 2025-12-04T12:13:24.2921288Z adding: test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-3223bbc58a6df58f.json (deflated 89%) 2025-12-04T12:13:24.2922904Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-c6dfa06eb71f87cd.json (deflated 91%) 2025-12-04T12:13:24.2924804Z adding: test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-8b65af0c5c32155a.json (deflated 81%) 2025-12-04T12:13:24.2926372Z adding: test/test-reports/python-pytest/distributed._tools.test_fake_collectives/distributed._tools.test_fake_collectives-619983be33386a60.json (deflated 40%) 2025-12-04T12:13:24.2928038Z adding: test/test-reports/python-pytest/distributed.test_control_collectives/distributed.test_control_collectives-3f2bbee5075aa12d.json (deflated 92%) 2025-12-04T12:13:24.2929521Z adding: test/test-reports/python-pytest/distributed.test_collective_utils/distributed.test_collective_utils-9394cde5cb4e06c2.json (deflated 88%) 2025-12-04T12:13:24.2931060Z adding: test/test-reports/python-pytest/distributed.test_c10d_object_collectives/distributed.test_c10d_object_collectives-d97a03c8ce93a326.json (deflated 90%) 2025-12-04T12:13:24.2932706Z adding: test/test-reports/python-pytest/distributed.tensor.debug.test_op_coverage/distributed.tensor.debug.test_op_coverage-ebb808e01daa256c.json (deflated 35%) 2025-12-04T12:13:24.2934429Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-3fe654a75937f34f.json (deflated 95%) 2025-12-04T12:13:24.2936184Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-2a8ca78a6e78af3b.json (deflated 79%) 2025-12-04T12:13:24.2937767Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-8d04d60499ed13c6.json (deflated 37%) 2025-12-04T12:13:24.2939349Z adding: test/test-reports/python-pytest/distributed.tensor.test_dtensor_compile/distributed.tensor.test_dtensor_compile-b72028e76a0d9d1d.json (deflated 90%) 2025-12-04T12:13:24.2940934Z adding: test/test-reports/python-pytest/distributed.test_aten_comm_compute_reordering/distributed.test_aten_comm_compute_reordering-7d1c62987df9c762.json (deflated 90%) 2025-12-04T12:13:24.2942474Z adding: test/test-reports/python-pytest/distributed._tools.test_sac_estimator/distributed._tools.test_sac_estimator-b0bd4af0f86e3ece.json (deflated 63%) 2025-12-04T12:13:24.2943944Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-c7b456a4f8311d2d.json (deflated 64%) 2025-12-04T12:13:24.2945585Z adding: test/test-reports/python-pytest/distributed.checkpoint._experimental.test_builder/distributed.checkpoint._experimental.test_builder-63eb721b9e0f0e0e.json (deflated 81%) 2025-12-04T12:13:24.2947380Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate_with_fsdp/distributed._composable.test_replicate_with_fsdp-4c94f165a9954ceb.json (deflated 83%) 2025-12-04T12:13:24.2949212Z adding: test/test-reports/python-pytest/distributed.tensor.test_xla_integration/distributed.tensor.test_xla_integration-953545bf454723f1.json (deflated 80%) 2025-12-04T12:13:24.2950886Z adding: test/test-reports/python-pytest/distributed.checkpoint._experimental.test_types/distributed.checkpoint._experimental.test_types-02c661f341e1f68e.json (deflated 75%) 2025-12-04T12:13:24.2952761Z adding: test/test-reports/python-pytest/distributed.tensor.experimental.test_register_sharding/distributed.tensor.experimental.test_register_sharding-ee24555d8c546070.json (deflated 76%) 2025-12-04T12:13:24.2954398Z adding: test/test-reports/python-pytest/distributed.test_backends/distributed.test_backends-11b04c9998663597.json (deflated 61%) 2025-12-04T12:13:24.2955858Z adding: test/test-reports/python-pytest/distributed.tensor.test_experimental_ops/distributed.tensor.test_experimental_ops-1d5fbe2b841d7120.json (deflated 81%) 2025-12-04T12:13:24.2957548Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_quantized_hf_storage/distributed.checkpoint.test_quantized_hf_storage-b35612bb520e65c5.json (deflated 64%) 2025-12-04T12:13:24.2959492Z adding: test/test-reports/python-pytest/distributed._composable.test_composability.test_pp_composability/distributed._composable.test_composability.test_pp_composability-6aa49750a7598206.json (deflated 97%) 2025-12-04T12:13:24.2961519Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_async_process_executor/distributed.checkpoint.test_async_process_executor-257bf63a3e72d6aa.json (deflated 82%) 2025-12-04T12:13:24.2963454Z adding: test/test-reports/python-pytest/distributed.tensor.test_tensor_ops/distributed.tensor.test_tensor_ops-dd61e7522cc11d1f.json (deflated 96%) 2025-12-04T12:13:24.2964830Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-2d037d076dad7e9e.json (deflated 94%) 2025-12-04T12:13:24.2966425Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_ignore_params/distributed._composable.fsdp.test_fully_shard_ignore_params-3bd5835b23ec8132.json (deflated 39%) 2025-12-04T12:13:24.2968329Z adding: test/test-reports/python-pytest/distributed.checkpoint._experimental.test_staging/distributed.checkpoint._experimental.test_staging-07a6a2de828c3109.json (deflated 83%) 2025-12-04T12:13:24.2970203Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion-7932544576c43c55.json (deflated 44%) 2025-12-04T12:13:24.2971901Z adding: test/test-reports/python-pytest/distributed.tensor.test_embedding_ops/distributed.tensor.test_embedding_ops-389c4439b22d140b.json (deflated 83%) 2025-12-04T12:13:24.2973531Z adding: test/test-reports/python-pytest/distributed.tensor.experimental.test_local_map/distributed.tensor.experimental.test_local_map-43637351f823edb4.json (deflated 86%) 2025-12-04T12:13:24.2975041Z adding: test/test-reports/python-pytest/distributed.test_local_tensor/distributed.test_local_tensor-2d16d75957767ef5.json (deflated 89%) 2025-12-04T12:13:24.2976580Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-f6baa7b934e7caab.json (deflated 84%) 2025-12-04T12:13:24.2978245Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_tp_checkpoint/distributed.checkpoint.test_tp_checkpoint-11d7a0d46082f98d.json (deflated 68%) 2025-12-04T12:13:24.2979772Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-4e9f4b21d1bcf8f6.json (deflated 86%) 2025-12-04T12:13:24.2981363Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_random_state/distributed.tensor.parallel.test_tp_random_state-3d1a164d85d827e3.json (deflated 39%) 2025-12-04T12:13:24.2982952Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_planner/distributed.checkpoint.test_planner-6c9d97ee371ac2d1.json (deflated 86%) 2025-12-04T12:13:24.2984562Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_checkpoint/distributed.checkpoint.test_dtensor_checkpoint-cb541eaa40da104f.json (deflated 43%) 2025-12-04T12:13:24.2986177Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule/distributed.pipelining.test_schedule-87ecb28af2fb8a27.json (deflated 94%) 2025-12-04T12:13:24.2988063Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-b60f2694fe7d2c52.json (deflated 67%) 2025-12-04T12:13:24.2989582Z adding: test/test-reports/python-pytest/distributed.test_run/distributed.test_run-ff2c644d612e7850.json (deflated 77%) 2025-12-04T12:13:24.2990945Z adding: test/test-reports/python-pytest/distributed.tensor.test_math_ops/distributed.tensor.test_math_ops-f7126468d0203b5f.json (deflated 94%) 2025-12-04T12:13:24.2992331Z adding: test/test-reports/python-pytest/distributed.fsdp.test_utils/distributed.fsdp.test_utils-62adb141232f9dae.json (deflated 82%) 2025-12-04T12:13:24.2993835Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-9094c50849cf3609.json (deflated 92%) 2025-12-04T12:13:24.2995609Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_compile/distributed._composable.fsdp.test_fully_shard_compile-0fa50fae7281ecc5.json (deflated 91%) 2025-12-04T12:13:24.2997432Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-db4a02d440fa1c43.json (deflated 74%) 2025-12-04T12:13:24.2999216Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_pg_transport/distributed.checkpoint.test_pg_transport-f8a475bbd4225371.json (deflated 88%) 2025-12-04T12:13:24.3001078Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-5f953c8bff259762.json (deflated 88%) 2025-12-04T12:13:24.3002802Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_utils/distributed.checkpoint.test_utils-3867a40d83a2b51a.json (deflated 90%) 2025-12-04T12:13:24.3004536Z adding: test/test-reports/python-pytest/distributed.checkpoint._experimental.test_checkpoint_process/distributed.checkpoint._experimental.test_checkpoint_process-8b0b149c7b33e94c.json (deflated 91%) 2025-12-04T12:13:24.3006232Z adding: test/test-reports/python-pytest/distributed.tensor.test_view_ops/distributed.tensor.test_view_ops-04bcb1d8735e8634.json (deflated 91%) 2025-12-04T12:13:24.3007695Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-0e669e7d48e3457f.json (deflated 97%) 2025-12-04T12:13:24.3009189Z adding: test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-e79a881124a4ef8b.json (deflated 82%) 2025-12-04T12:13:24.3010866Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-8e7248bfdb62ae56.json (deflated 87%) 2025-12-04T12:13:24.3012678Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint_cpu/distributed.checkpoint.test_file_system_checkpoint_cpu-86828f53aae09692.json (deflated 87%) 2025-12-04T12:13:24.3014330Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm_hooks/distributed.fsdp.test_fsdp_comm_hooks-e30f0f398ca36b1a.json (deflated 95%) 2025-12-04T12:13:24.3016234Z adding: test/test-reports/python-pytest/distributed._shard.test_sharder/distributed._shard.test_sharder-c376df3131abbe36.json (deflated 65%) 2025-12-04T12:13:24.3017786Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-130017345592955f.json (deflated 84%) 2025-12-04T12:13:24.3019905Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-4367c9892b5119af.json (deflated 75%) 2025-12-04T12:13:24.3021770Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-57a88f260c958571.json (deflated 93%) 2025-12-04T12:13:24.3023415Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5c2225b883027b57.json (deflated 33%) 2025-12-04T12:13:24.3026255Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e989bf90bb499159.json (deflated 32%) 2025-12-04T12:13:24.3027688Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5358d199d91e916e.json (deflated 31%) 2025-12-04T12:13:24.3029013Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e8f01c1258f32697.json (deflated 31%) 2025-12-04T12:13:24.3030340Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-ac92f4ba57cf76c4.json (deflated 32%) 2025-12-04T12:13:24.3031657Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-eda4e22775599b5a.json (deflated 31%) 2025-12-04T12:13:24.3032977Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-f2042aa2d074e648.json (deflated 32%) 2025-12-04T12:13:24.3034308Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-eace822c5c4d90b0.json (deflated 33%) 2025-12-04T12:13:24.3035770Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e2aaf0e6db1856f6.json (deflated 33%) 2025-12-04T12:13:24.3037087Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-aba1e98d5f917e6f.json (deflated 34%) 2025-12-04T12:13:24.3038414Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-c8752499cbdebb6b.json (deflated 33%) 2025-12-04T12:13:24.3039881Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e365ef3c3de4e526.json (deflated 33%) 2025-12-04T12:13:24.3041154Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-3d0c386b38968978.json (deflated 32%) 2025-12-04T12:13:24.3042434Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-c9893cb7ff9eb7cb.json (deflated 33%) 2025-12-04T12:13:24.3043758Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-d8ca55d72dd5d963.json (deflated 34%) 2025-12-04T12:13:24.3045040Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-d6cc878cd0b0d21e.json (deflated 33%) 2025-12-04T12:13:24.3046319Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5399ee4967a63bf6.json (deflated 32%) 2025-12-04T12:13:24.3047740Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-97a90e2322241ed5.json (stored 0%) 2025-12-04T12:13:24.3049400Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_binary_cmp/distributed._shard.sharded_tensor.ops.test_binary_cmp-1b60820ad9df53ea.json (deflated 81%) 2025-12-04T12:13:24.3051011Z adding: test/test-reports/python-pytest/distributed.nn.jit.test_instantiator/distributed.nn.jit.test_instantiator-26bb23415f1ce435.json (deflated 74%) 2025-12-04T12:13:24.3076319Z adding: test/test-reports/python-pytest/distributed._shard.sharding_spec.test_sharding_spec/distributed._shard.sharding_spec.test_sharding_spec-b7677c73b05bb97d.json (deflated 90%) 2025-12-04T12:13:24.3077877Z adding: test/test-reports/python-pytest/distributed.test_nccl/distributed.test_nccl-6ab9552fe0bfe15f.json (deflated 87%) 2025-12-04T12:13:24.3079197Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_misc/distributed.fsdp.test_fsdp_misc-ff29f8b606fc4847.json (deflated 86%) 2025-12-04T12:13:24.3080683Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_meta/distributed.fsdp.test_fsdp_meta-8050e6a6d02fe535.json (deflated 92%) 2025-12-04T12:13:24.3082033Z adding: test/test-reports/python-pytest/distributed.test_data_parallel/distributed.test_data_parallel-c1f5c0fcbf415006.json (deflated 88%) 2025-12-04T12:13:24.3083506Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e8d73f4591d62db9.json (deflated 35%) 2025-12-04T12:13:24.3085091Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9a221c145b1a74c2.json (deflated 35%) 2025-12-04T12:13:24.3086670Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-08cb4ec451a35d57.json (deflated 37%) 2025-12-04T12:13:24.3088247Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dee1ac91d0c754d6.json (deflated 36%) 2025-12-04T12:13:24.3089831Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-350598816b45384d.json (deflated 39%) 2025-12-04T12:13:24.3091412Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-510c0213c3d43ce4.json (deflated 38%) 2025-12-04T12:13:24.3097343Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b9f09d55835d87af.json (deflated 38%) 2025-12-04T12:13:24.3098945Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ed820609834e1378.json (deflated 39%) 2025-12-04T12:13:24.3100555Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c88e834a8b4dbb20.json (deflated 37%) 2025-12-04T12:13:24.3102157Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f0b18dfb3a59dad1.json (deflated 37%) 2025-12-04T12:13:24.3103747Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4743d357d297f6dc.json (deflated 37%) 2025-12-04T12:13:24.3105357Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4662c39cb6a4b928.json (deflated 38%) 2025-12-04T12:13:24.3106922Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-73028dd94dff8c49.json (deflated 37%) 2025-12-04T12:13:24.3108773Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3ee7cb942a2cc6e3.json (deflated 38%) 2025-12-04T12:13:24.3110433Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5669ef320dd02577.json (deflated 38%) 2025-12-04T12:13:24.3112037Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c3e4151039d68fa8.json (deflated 38%) 2025-12-04T12:13:24.3113649Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-024c7a32a248042e.json (deflated 37%) 2025-12-04T12:13:24.3115258Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c97e92ee7b028ae3.json (deflated 36%) 2025-12-04T12:13:24.3116866Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d8dd34d3359043f5.json (deflated 37%) 2025-12-04T12:13:24.3118473Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-934a5e5071fe9c36.json (deflated 37%) 2025-12-04T12:13:24.3120299Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d42c1866d87ab3db.json (deflated 36%) 2025-12-04T12:13:24.3121879Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d46cf2452bc01536.json (deflated 36%) 2025-12-04T12:13:24.3123469Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8f4f39fa980e82f2.json (deflated 37%) 2025-12-04T12:13:24.3125434Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0652764081d6da21.json (deflated 37%) 2025-12-04T12:13:24.3127048Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-610c219c10381302.json (deflated 38%) 2025-12-04T12:13:24.3128668Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ff765bddb1f8ea71.json (deflated 37%) 2025-12-04T12:13:24.3130299Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9be5a3fd2574b057.json (deflated 38%) 2025-12-04T12:13:24.3132058Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-df5bf940ae0a6258.json (deflated 38%) 2025-12-04T12:13:24.3133670Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a70e67bbc896acc2.json (deflated 38%) 2025-12-04T12:13:24.3135297Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e5a6bcea490b9fda.json (deflated 38%) 2025-12-04T12:13:24.3137184Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1d02ec45c02f60fe.json (deflated 36%) 2025-12-04T12:13:24.3138756Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-589f53701a6c7b6a.json (deflated 37%) 2025-12-04T12:13:24.3140377Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-37a88acccbf1ecd0.json (deflated 37%) 2025-12-04T12:13:24.3141940Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9e0e61cd971b69bb.json (deflated 37%) 2025-12-04T12:13:24.3143513Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a25ed88c721098f2.json (deflated 37%) 2025-12-04T12:13:24.3145085Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c9d57215b8ca5d9f.json (deflated 38%) 2025-12-04T12:13:24.3146661Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-07a2c7293983a21f.json (deflated 38%) 2025-12-04T12:13:24.3148469Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e7d26bc200bae427.json (deflated 36%) 2025-12-04T12:13:24.3150171Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c4303c90119175e3.json (deflated 36%) 2025-12-04T12:13:24.3151787Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b08f37975720f6b4.json (deflated 37%) 2025-12-04T12:13:24.3153408Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-75b604dc9c04c4c7.json (deflated 37%) 2025-12-04T12:13:24.3155035Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8dc1d7aea93bdb34.json (deflated 37%) 2025-12-04T12:13:24.3156654Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4a3867d6b6932f10.json (deflated 37%) 2025-12-04T12:13:24.3158277Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0318829c01be08fa.json (deflated 37%) 2025-12-04T12:13:24.3159999Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-693b8cadac47d9c2.json (deflated 37%) 2025-12-04T12:13:24.3161571Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0ecc816e8885b9bc.json (deflated 37%) 2025-12-04T12:13:24.3163260Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d3419121e211fb91.json (deflated 37%) 2025-12-04T12:13:24.3164822Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2c51cc1e26d5acf3.json (deflated 37%) 2025-12-04T12:13:24.3166467Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4746fedc7b470508.json (deflated 37%) 2025-12-04T12:13:24.3168063Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d82b54717ce755b5.json (deflated 37%) 2025-12-04T12:13:24.3169634Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-730ab8cf8f5003b5.json (deflated 38%) 2025-12-04T12:13:24.3171223Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ef621cb706a80f36.json (deflated 37%) 2025-12-04T12:13:24.3172965Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_unshard_params/distributed.fsdp.test_fsdp_unshard_params-49f6028e4feadb15.json (deflated 91%) 2025-12-04T12:13:24.3174609Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-b148a45a231b8de9.json (deflated 77%) 2025-12-04T12:13:24.3176345Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding_bag/distributed._shard.sharded_tensor.ops.test_embedding_bag-0fb42e6113dec695.json (deflated 68%) 2025-12-04T12:13:24.3177934Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e3270986f520c455.json (deflated 37%) 2025-12-04T12:13:24.3179295Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9f07ad58b8cd3f14.json (deflated 37%) 2025-12-04T12:13:24.3180658Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-38ae8a6c9446176c.json (deflated 37%) 2025-12-04T12:13:24.3182102Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9be7dc10fc456dd4.json (deflated 36%) 2025-12-04T12:13:24.3183429Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a4bb8d2a63c3e77a.json (deflated 36%) 2025-12-04T12:13:24.3184729Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a65cb5796fd83f18.json (deflated 36%) 2025-12-04T12:13:24.3186118Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-44d720723f83b050.json (deflated 36%) 2025-12-04T12:13:24.3187531Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a0db0c7d2253e897.json (deflated 36%) 2025-12-04T12:13:24.3189097Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-4d41a04cacb8591f.json (deflated 36%) 2025-12-04T12:13:24.3190501Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-79c50cf35aedde1f.json (deflated 36%) 2025-12-04T12:13:24.3191905Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e36163ffaa730c12.json (deflated 36%) 2025-12-04T12:13:24.3193398Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_use_orig_params/distributed.fsdp.test_fsdp_use_orig_params-c1df23351bfa5ee8.json (deflated 90%) 2025-12-04T12:13:24.3194861Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-093e5ad7fc260f19.json (deflated 35%) 2025-12-04T12:13:24.3196194Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-60dcb39f38cfdee4.json (deflated 35%) 2025-12-04T12:13:24.3197518Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-c9c5102c76f0a9a2.json (deflated 34%) 2025-12-04T12:13:24.3198937Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-312bbf42e62ed9b6.json (deflated 34%) 2025-12-04T12:13:24.3200424Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-75c9bd860618416d.json (deflated 35%) 2025-12-04T12:13:24.3201675Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-104493b7b71c2fe7.json (deflated 34%) 2025-12-04T12:13:24.3202904Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-d6965728c111189f.json (deflated 34%) 2025-12-04T12:13:24.3204195Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-550261c769de2900.json (deflated 34%) 2025-12-04T12:13:24.3205441Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-c61f939b9c1212d0.json (deflated 34%) 2025-12-04T12:13:24.3206691Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-f26443fa9fd7d2be.json (deflated 33%) 2025-12-04T12:13:24.3207974Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-33f9aeac8ff2d88d.json (deflated 34%) 2025-12-04T12:13:24.3209208Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-6f219815bc6782ef.json (deflated 34%) 2025-12-04T12:13:24.3210455Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-1f65ab45ec69dc91.json (deflated 34%) 2025-12-04T12:13:24.3211702Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a7ef02fe48ea3a40.json (deflated 32%) 2025-12-04T12:13:24.3212943Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a8622e73769262d0.json (deflated 34%) 2025-12-04T12:13:24.3214294Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-3ba3606e730f6b82.json (deflated 34%) 2025-12-04T12:13:24.3215545Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-48cf7919df9353bf.json (deflated 33%) 2025-12-04T12:13:24.3216789Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-5927afc313986450.json (deflated 33%) 2025-12-04T12:13:24.3218034Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-e564db12bfa2b80c.json (deflated 33%) 2025-12-04T12:13:24.3219271Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-17d2371d605f4c7c.json (deflated 35%) 2025-12-04T12:13:24.3220520Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b0038959c806a90a.json (deflated 35%) 2025-12-04T12:13:24.3221763Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-5cc7624c76b4d82c.json (deflated 35%) 2025-12-04T12:13:24.3223016Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-ef399536ef6f172f.json (deflated 35%) 2025-12-04T12:13:24.3224738Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b77e93b7526c6e31.json (deflated 35%) 2025-12-04T12:13:24.3226071Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-52e140e51a31b98a.json (deflated 34%) 2025-12-04T12:13:24.3227475Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b7bc54ed87c9274a.json (deflated 33%) 2025-12-04T12:13:24.3228813Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-927eadde7027cdb3.json (deflated 32%) 2025-12-04T12:13:24.3230273Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_mixed_precision/distributed.fsdp.test_fsdp_mixed_precision-c50711adaece2e48.json (deflated 96%) 2025-12-04T12:13:24.3231847Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-81f85fc9fa3db30b.json (deflated 36%) 2025-12-04T12:13:24.3233136Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0df1dc2a08194e48.json (deflated 33%) 2025-12-04T12:13:24.3234415Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-43498c0a89946df9.json (deflated 33%) 2025-12-04T12:13:24.3235722Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-af4c89e072116764.json (deflated 33%) 2025-12-04T12:13:24.3236999Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-98f78a29e90359ed.json (deflated 33%) 2025-12-04T12:13:24.3238279Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-30adfd6c7b8ecf71.json (deflated 32%) 2025-12-04T12:13:24.3239790Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-581dd59091205273.json (deflated 33%) 2025-12-04T12:13:24.3240988Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d528c59f12482b1.json (deflated 33%) 2025-12-04T12:13:24.3242188Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d6315101734e178a.json (deflated 57%) 2025-12-04T12:13:24.3243402Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d89c8d6fc7b73a1b.json (deflated 57%) 2025-12-04T12:13:24.3244608Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c23cf3f46128ac0f.json (deflated 33%) 2025-12-04T12:13:24.3245818Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3a09c39260c6bdcb.json (deflated 33%) 2025-12-04T12:13:24.3247012Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f486758b2548beb0.json (deflated 33%) 2025-12-04T12:13:24.3248222Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-36d76263f3b24111.json (deflated 33%) 2025-12-04T12:13:24.3249424Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f1c94044951be399.json (deflated 33%) 2025-12-04T12:13:24.3250623Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b9df70724e6dcd35.json (deflated 33%) 2025-12-04T12:13:24.3251821Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fbd78536cd21cf97.json (deflated 34%) 2025-12-04T12:13:24.3253309Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ca3fc647503e2182.json (deflated 33%) 2025-12-04T12:13:24.3254553Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-15e1be913d8837e9.json (deflated 33%) 2025-12-04T12:13:24.3255803Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8a7b7a30260424a5.json (deflated 33%) 2025-12-04T12:13:24.3257034Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-13f85e6fb9609878.json (deflated 35%) 2025-12-04T12:13:24.3258279Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d09ed72d675bfdc8.json (deflated 35%) 2025-12-04T12:13:24.3259526Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3eb78e74581a557b.json (deflated 36%) 2025-12-04T12:13:24.3260764Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-90acb82834a56757.json (deflated 35%) 2025-12-04T12:13:24.3261991Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a1738cfdeff44f2e.json (deflated 35%) 2025-12-04T12:13:24.3263557Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7936e4b0274d7ad6.json (deflated 34%) 2025-12-04T12:13:24.3264921Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bd821bd95ef10bd6.json (deflated 34%) 2025-12-04T12:13:24.3266142Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5839fc6b3294dc3b.json (deflated 36%) 2025-12-04T12:13:24.3267413Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-be22aa9f1a9dc572.json (deflated 36%) 2025-12-04T12:13:24.3268882Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-63d98e3d16f90c98.json (deflated 35%) 2025-12-04T12:13:24.3270165Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-93df4bdbf125060c.json (deflated 35%) 2025-12-04T12:13:24.3271448Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c1c3a483347ae16e.json (deflated 35%) 2025-12-04T12:13:24.3272772Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5c6d35f4b8178e66.json (deflated 36%) 2025-12-04T12:13:24.3274040Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f4a815301947af44.json (deflated 34%) 2025-12-04T12:13:24.3275321Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ea05d17fa83aaab3.json (deflated 35%) 2025-12-04T12:13:24.3276607Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c99c7919907a8f51.json (deflated 35%) 2025-12-04T12:13:24.3277890Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3228075a6adb947.json (deflated 37%) 2025-12-04T12:13:24.3279174Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5f3dfac751b51f15.json (deflated 35%) 2025-12-04T12:13:24.3280642Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9836e36a022fec7d.json (deflated 37%) 2025-12-04T12:13:24.3281852Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-481f5a9e007c92bc.json (deflated 37%) 2025-12-04T12:13:24.3283058Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8e5333f0f8c7af12.json (deflated 35%) 2025-12-04T12:13:24.3284254Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-27c65021619c5afb.json (deflated 35%) 2025-12-04T12:13:24.3285466Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3bc159f1fc89dbaa.json (deflated 33%) 2025-12-04T12:13:24.3286671Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4183a04f60a93385.json (deflated 44%) 2025-12-04T12:13:24.3287881Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b6ee0414e522cc1e.json (deflated 44%) 2025-12-04T12:13:24.3289082Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ba060feb7f6eb8f7.json (deflated 35%) 2025-12-04T12:13:24.3290287Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3aa486503236fa6.json (deflated 35%) 2025-12-04T12:13:24.3291488Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a218cf86ddf03214.json (deflated 35%) 2025-12-04T12:13:24.3292702Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fb621eae387ef1df.json (deflated 35%) 2025-12-04T12:13:24.3293916Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-80e6fc2adcb64c0a.json (deflated 35%) 2025-12-04T12:13:24.3295114Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2323c2353e236ae2.json (deflated 35%) 2025-12-04T12:13:24.3296382Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-21c4771a3c9fd2ac.json (deflated 34%) 2025-12-04T12:13:24.3297772Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e704bc72dcc2db74.json (deflated 35%) 2025-12-04T12:13:24.3299018Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8da80ebf3429571e.json (deflated 34%) 2025-12-04T12:13:24.3300285Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-aad331620921f562.json (deflated 34%) 2025-12-04T12:13:24.3301529Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4d876218f2c25b17.json (deflated 35%) 2025-12-04T12:13:24.3302771Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8a4e259673457968.json (deflated 32%) 2025-12-04T12:13:24.3304053Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d2017d482dc7eebb.json (deflated 33%) 2025-12-04T12:13:24.3305289Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cd8336cfdb34c878.json (deflated 36%) 2025-12-04T12:13:24.3306533Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-be2011792fadd5cd.json (deflated 33%) 2025-12-04T12:13:24.3308036Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fd2ba77f6fee2f8a.json (deflated 34%) 2025-12-04T12:13:24.3309326Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-da6ff71bbbbcfd5b.json (deflated 33%) 2025-12-04T12:13:24.3310599Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-00c7457a094406e2.json (deflated 33%) 2025-12-04T12:13:24.3311886Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f38ba7e725b3376c.json (deflated 33%) 2025-12-04T12:13:24.3313170Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a85d18d4001a0d1f.json (deflated 34%) 2025-12-04T12:13:24.3314454Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-77e675aec14c1ef5.json (deflated 35%) 2025-12-04T12:13:24.3315728Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-679877f4d338af19.json (deflated 36%) 2025-12-04T12:13:24.3317010Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8d26022f0d5f0e5b.json (deflated 34%) 2025-12-04T12:13:24.3318294Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d52baf36617de6c.json (deflated 33%) 2025-12-04T12:13:24.3319697Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-97cfc974aeae8bab.json (deflated 33%) 2025-12-04T12:13:24.3321032Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dc857ca5b8949f50.json (deflated 34%) 2025-12-04T12:13:24.3322240Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-68d634b03e6f1cf2.json (deflated 34%) 2025-12-04T12:13:24.3323448Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-797d0b27326bd9f0.json (deflated 33%) 2025-12-04T12:13:24.3325312Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7f4f2591d711a574.json (deflated 34%) 2025-12-04T12:13:24.3326580Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-54ee371b40f31a9b.json (deflated 33%) 2025-12-04T12:13:24.3327868Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bc7d0e9f12eb7a6c.json (deflated 32%) 2025-12-04T12:13:24.3329294Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4b8aea19c365479d.json (deflated 33%) 2025-12-04T12:13:24.3330584Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ace037ae618597c7.json (deflated 33%) 2025-12-04T12:13:24.3331864Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8980b265f0dcd572.json (deflated 32%) 2025-12-04T12:13:24.3333183Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2f6c447cb3d8d9ba.json (deflated 32%) 2025-12-04T12:13:24.3334471Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-902c7c5c4fac1c79.json (deflated 32%) 2025-12-04T12:13:24.3335756Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3399e4ec5bb86d0f.json (deflated 35%) 2025-12-04T12:13:24.3337277Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ee7e5984f1b0da3e.json (deflated 41%) 2025-12-04T12:13:24.3338805Z adding: test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-daac7fd3db05a588.json (stored 0%) 2025-12-04T12:13:24.3340661Z adding: test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-a07a6d6cccb031bb.json (stored 0%) 2025-12-04T12:13:24.3342511Z adding: test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-7c0282eac181e46a.json (stored 0%) 2025-12-04T12:13:24.3344131Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-975ac1cd2ed3ea0d.json (stored 0%) 2025-12-04T12:13:24.3345526Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8babbac9d40e37c7.json (stored 0%) 2025-12-04T12:13:24.3346905Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4f717ad1d99b753.json (stored 0%) 2025-12-04T12:13:24.3348558Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85365fcd3056dbf4.json (stored 0%) 2025-12-04T12:13:24.3350036Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af2626afcb0e4552.json (stored 0%) 2025-12-04T12:13:24.3351508Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a30a6d9cd45f8b4.json (stored 0%) 2025-12-04T12:13:24.3352972Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6f6e95ed9bc43f3.json (stored 0%) 2025-12-04T12:13:24.3354452Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17e902b4247dcc9d.json (stored 0%) 2025-12-04T12:13:24.3355925Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a3b3a8e58fbb071.json (stored 0%) 2025-12-04T12:13:24.3357638Z adding: test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-69522635455745bd.json (stored 0%) 2025-12-04T12:13:24.3359719Z adding: test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-b1c2edd3b83dd6a9.json (stored 0%) 2025-12-04T12:13:24.3361621Z adding: test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-8fc6dea04b4e72e8.json (stored 0%) 2025-12-04T12:13:24.3363437Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-337a55931fe74487.json (stored 0%) 2025-12-04T12:13:24.3364832Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90b3763c34aa2960.json (stored 0%) 2025-12-04T12:13:24.3366223Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d0f38b19d35d9f0.json (stored 0%) 2025-12-04T12:13:24.3367627Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62bde0ef8091e7c6.json (stored 0%) 2025-12-04T12:13:24.3369020Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-101e5756265aa78e.json (stored 0%) 2025-12-04T12:13:24.3370410Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3553a63d7be9bc12.json (stored 0%) 2025-12-04T12:13:24.3371838Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c18a85b55f10e02.json (stored 0%) 2025-12-04T12:13:24.3373227Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee7c4f0b0fe742e4.json (stored 0%) 2025-12-04T12:13:24.3374598Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a74d4ada0231688.json (stored 0%) 2025-12-04T12:13:24.3376238Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-694fabcdc25b908f.json (deflated 51%) 2025-12-04T12:13:24.3378162Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cc8c47d77c37f668.json (deflated 51%) 2025-12-04T12:13:24.3380151Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-93164f6c3fb62a8d.json (deflated 41%) 2025-12-04T12:13:24.3382016Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-55976768b0d94051.json (deflated 41%) 2025-12-04T12:13:24.3383894Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-ed34111b302d3808.json (deflated 41%) 2025-12-04T12:13:24.3385784Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-160b76a599e6e82f.json (deflated 41%) 2025-12-04T12:13:24.3387500Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a57a7c1f6c446d8.json (deflated 38%) 2025-12-04T12:13:24.3389176Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9a464945bf1177b.json (deflated 39%) 2025-12-04T12:13:24.3390683Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1497afbb3545da85.json (deflated 42%) 2025-12-04T12:13:24.3392195Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cbb5e6acdeef21a7.json (deflated 47%) 2025-12-04T12:13:24.3393715Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e37edf773eeec4cb.json (deflated 43%) 2025-12-04T12:13:24.3395232Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9cafccc700b8925.json (deflated 43%) 2025-12-04T12:13:24.3396739Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b34ce6e907f1d03.json (deflated 43%) 2025-12-04T12:13:24.3398298Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1450c3d527670c7.json (deflated 37%) 2025-12-04T12:13:24.3400013Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85297e8464ee07d1.json (deflated 44%) 2025-12-04T12:13:24.3401466Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aff79cf3432fbb8f.json (deflated 44%) 2025-12-04T12:13:24.3403083Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8141293960cc76b4.json (deflated 37%) 2025-12-04T12:13:24.3404528Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ed1d3854cb079b5.json (deflated 37%) 2025-12-04T12:13:24.3406030Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f5ee7b2c181b77f.json (deflated 37%) 2025-12-04T12:13:24.3407571Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3de0668036a4d46a.json (deflated 37%) 2025-12-04T12:13:24.3408989Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ed12ff4e601d4a5.json (deflated 49%) 2025-12-04T12:13:24.3410410Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55742e93980fa69f.json (deflated 37%) 2025-12-04T12:13:24.3411820Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d1649152d68be9c.json (deflated 37%) 2025-12-04T12:13:24.3413245Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3c70db6f7d0cc8f.json (deflated 37%) 2025-12-04T12:13:24.3414670Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0fb83c60b047974.json (deflated 47%) 2025-12-04T12:13:24.3416090Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78a8a7c12b3fdbcd.json (deflated 37%) 2025-12-04T12:13:24.3417492Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88531b1bf42e91d7.json (deflated 37%) 2025-12-04T12:13:24.3418904Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22e23e45bc80123f.json (deflated 38%) 2025-12-04T12:13:24.3420315Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f92f6113f8978240.json (deflated 38%) 2025-12-04T12:13:24.3421732Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15dc7f75169494c0.json (deflated 36%) 2025-12-04T12:13:24.3423133Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e73ef2b0e35926a2.json (deflated 37%) 2025-12-04T12:13:24.3425070Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f598c7195f78ef8.json (deflated 37%) 2025-12-04T12:13:24.3426583Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88b5e238fde7cdb9.json (deflated 37%) 2025-12-04T12:13:24.3428182Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8880a41f88a6bb0.json (deflated 46%) 2025-12-04T12:13:24.3429696Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b0298f7abe568d4.json (deflated 57%) 2025-12-04T12:13:24.3431338Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e4e163fd1daa635.json (deflated 36%) 2025-12-04T12:13:24.3432857Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9ab104c2f6083ce.json (deflated 46%) 2025-12-04T12:13:24.3434366Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d76d59daf2e605c7.json (deflated 46%) 2025-12-04T12:13:24.3435906Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c31c06f41345c5c.json (deflated 43%) 2025-12-04T12:13:24.3437399Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a98e316dc116cc1.json (deflated 36%) 2025-12-04T12:13:24.3438914Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-723af9187a42bd99.json (deflated 37%) 2025-12-04T12:13:24.3440502Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf92d20898f195ab.json (deflated 43%) 2025-12-04T12:13:24.3441920Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-524dc1dcdc5a1168.json (deflated 43%) 2025-12-04T12:13:24.3443338Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98d2197e9fdc8948.json (deflated 37%) 2025-12-04T12:13:24.3444732Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2102f231775de0a.json (deflated 45%) 2025-12-04T12:13:24.3446421Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-271cbb7ef1d534a9.json (deflated 44%) 2025-12-04T12:13:24.3447837Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d42c353b826e766.json (deflated 37%) 2025-12-04T12:13:24.3449255Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4f9face38efde3d.json (deflated 37%) 2025-12-04T12:13:24.3450670Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65eb8dabd26f049f.json (deflated 37%) 2025-12-04T12:13:24.3452093Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-993c00909ddd3eaa.json (deflated 37%) 2025-12-04T12:13:24.3453508Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a20300b26842835f.json (deflated 36%) 2025-12-04T12:13:24.3454917Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8300341a4195c4a2.json (deflated 37%) 2025-12-04T12:13:24.3456328Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e45287a9f93d9858.json (deflated 37%) 2025-12-04T12:13:24.3457728Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa4ae74c761f06da.json (deflated 38%) 2025-12-04T12:13:24.3459148Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b7742dab30c5a9.json (deflated 36%) 2025-12-04T12:13:24.3460623Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92706bcf8036c15a.json (deflated 37%) 2025-12-04T12:13:24.3461963Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e2d29ed0b71edf9.json (deflated 36%) 2025-12-04T12:13:24.3463567Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b82693979bf3f797.json (deflated 36%) 2025-12-04T12:13:24.3464982Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40f956e1991c429b.json (deflated 37%) 2025-12-04T12:13:24.3466414Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be8fbd355ae7d4d2.json (deflated 36%) 2025-12-04T12:13:24.3468105Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00fdf0a71d560046.json (deflated 38%) 2025-12-04T12:13:24.3469610Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f23f45b0fd1b4e0.json (deflated 37%) 2025-12-04T12:13:24.3471102Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a704bee28922d105.json (deflated 44%) 2025-12-04T12:13:24.3472648Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59aa1d9ad1b952bc.json (deflated 44%) 2025-12-04T12:13:24.3474157Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f46bcad2881001f4.json (deflated 36%) 2025-12-04T12:13:24.3475661Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bdf0f48b15d753a.json (deflated 43%) 2025-12-04T12:13:24.3477153Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3701eb2bf6e59b5c.json (deflated 43%) 2025-12-04T12:13:24.3478666Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4294fcb622b36e12.json (deflated 37%) 2025-12-04T12:13:24.3480236Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25f21f2c9cc3f58e.json (deflated 43%) 2025-12-04T12:13:24.3481816Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a888dfec83025ca2.json (deflated 45%) 2025-12-04T12:13:24.3483242Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a2971cbec516283.json (deflated 46%) 2025-12-04T12:13:24.3484653Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-831b908a573477b4.json (deflated 36%) 2025-12-04T12:13:24.3486071Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-838476cdd85e63de.json (deflated 38%) 2025-12-04T12:13:24.3487487Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7c1e5f9052a62ed.json (deflated 45%) 2025-12-04T12:13:24.3488924Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9fe4918eb2ec3fc.json (deflated 44%) 2025-12-04T12:13:24.3490334Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18ee4fc76444d2ea.json (deflated 37%) 2025-12-04T12:13:24.3491746Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf306b33274dddf6.json (deflated 44%) 2025-12-04T12:13:24.3493158Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c91a26250cab8d8.json (deflated 37%) 2025-12-04T12:13:24.3494688Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12d1ac9616f3d37b.json (deflated 37%) 2025-12-04T12:13:24.3496225Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24baa892d95df88d.json (deflated 44%) 2025-12-04T12:13:24.3497683Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1238a6b3de9abf1.json (deflated 43%) 2025-12-04T12:13:24.3499100Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5960ed309c5523d7.json (deflated 37%) 2025-12-04T12:13:24.3500534Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-484220487412d630.json (deflated 37%) 2025-12-04T12:13:24.3501952Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcdabce46aafa3d7.json (deflated 37%) 2025-12-04T12:13:24.3503356Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-712b9444633e7909.json (deflated 38%) 2025-12-04T12:13:24.3504801Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22f63a5524b50ec1.json (deflated 44%) 2025-12-04T12:13:24.3506216Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9abdeaa3f668748b.json (deflated 37%) 2025-12-04T12:13:24.3507872Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db023d74613c38df.json (deflated 44%) 2025-12-04T12:13:24.3509385Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d349e5cd46aba4b9.json (deflated 44%) 2025-12-04T12:13:24.3510881Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dcab0e823bbcb484.json (deflated 57%) 2025-12-04T12:13:24.3512387Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-107117b39386c513.json (deflated 36%) 2025-12-04T12:13:24.3513894Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2a16f55ac9cd547.json (deflated 43%) 2025-12-04T12:13:24.3515394Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f920c9d47a13120.json (deflated 43%) 2025-12-04T12:13:24.3516880Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0adedc6075462154.json (deflated 43%) 2025-12-04T12:13:24.3518386Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51b3a60aa7e4ef26.json (deflated 45%) 2025-12-04T12:13:24.3519987Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9abab74452896355.json (deflated 37%) 2025-12-04T12:13:24.3521456Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa158d78c6a1047c.json (deflated 37%) 2025-12-04T12:13:24.3523018Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-dbafcff6c2749b84.json (deflated 51%) 2025-12-04T12:13:24.3525248Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-80abfc2e08c5c31a.json (deflated 51%) 2025-12-04T12:13:24.3527269Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-281d630b2fcf32af.json (deflated 41%) 2025-12-04T12:13:24.3529289Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-875849d617d7b51e.json (deflated 41%) 2025-12-04T12:13:24.3531409Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-4cfe0d1200d89b83.json (deflated 41%) 2025-12-04T12:13:24.3533421Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-366840efd0ab41ce.json (deflated 41%) 2025-12-04T12:13:24.3535180Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-365036945f26d41b.json (deflated 38%) 2025-12-04T12:13:24.3536935Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e819ba4cf6e31f8.json (deflated 39%) 2025-12-04T12:13:24.3538279Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40bc39b1652660d2.json (deflated 42%) 2025-12-04T12:13:24.3539669Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f9afae11c179070.json (deflated 47%) 2025-12-04T12:13:24.3540997Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3c746330754bbc2.json (deflated 43%) 2025-12-04T12:13:24.3542337Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b68ead304b495067.json (deflated 43%) 2025-12-04T12:13:24.3543683Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7886ccd5b9eeb78.json (deflated 43%) 2025-12-04T12:13:24.3545031Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21598fac9ec33b36.json (deflated 37%) 2025-12-04T12:13:24.3546372Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45fc07c2582c64de.json (deflated 44%) 2025-12-04T12:13:24.3547948Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9a64b3d5828bdd7.json (deflated 44%) 2025-12-04T12:13:24.3549459Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f441409f584e7860.json (deflated 37%) 2025-12-04T12:13:24.3550965Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02d51cd92aaffd99.json (deflated 37%) 2025-12-04T12:13:24.3552485Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd7c20c72a58c490.json (deflated 37%) 2025-12-04T12:13:24.3553989Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a885f907c0ea15f.json (deflated 37%) 2025-12-04T12:13:24.3555506Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd6772c504021b1c.json (deflated 49%) 2025-12-04T12:13:24.3557018Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b129e835aba47f3.json (deflated 37%) 2025-12-04T12:13:24.3558533Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dca90ceb118f7f51.json (deflated 37%) 2025-12-04T12:13:24.3560157Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4c7cdb746d63750.json (deflated 37%) 2025-12-04T12:13:24.3561489Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b62455b26fd75a3.json (deflated 47%) 2025-12-04T12:13:24.3562828Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c6c3445d3489ed7.json (deflated 37%) 2025-12-04T12:13:24.3564234Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc9043c9b1f5531d.json (deflated 37%) 2025-12-04T12:13:24.3565582Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fe0c34ac3a70b57.json (deflated 38%) 2025-12-04T12:13:24.3566911Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93b994665f730626.json (deflated 38%) 2025-12-04T12:13:24.3568273Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e40c62350b42648a.json (deflated 36%) 2025-12-04T12:13:24.3569608Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c79494752e7d5ea.json (deflated 37%) 2025-12-04T12:13:24.3570949Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6369d472a2cdae13.json (deflated 37%) 2025-12-04T12:13:24.3572335Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-342afb36b7c5d09b.json (deflated 37%) 2025-12-04T12:13:24.3573661Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f18aaac9bd89cd2c.json (deflated 46%) 2025-12-04T12:13:24.3575010Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2615b1994fa3617.json (deflated 57%) 2025-12-04T12:13:24.3576343Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e62634fc743f9964.json (deflated 36%) 2025-12-04T12:13:24.3577679Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11b374627b30152b.json (deflated 46%) 2025-12-04T12:13:24.3579012Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2922c97c2127de2.json (deflated 46%) 2025-12-04T12:13:24.3580355Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25e6eb10dce656e7.json (deflated 43%) 2025-12-04T12:13:24.3581705Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c2f928358ecde73.json (deflated 37%) 2025-12-04T12:13:24.3583046Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d84b93ba23264de.json (deflated 37%) 2025-12-04T12:13:24.3584391Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48132f4775de3019.json (deflated 43%) 2025-12-04T12:13:24.3585739Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4552be91feabceb2.json (deflated 43%) 2025-12-04T12:13:24.3587099Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6475011a4c42f67b.json (deflated 37%) 2025-12-04T12:13:24.3588780Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d893aced2165d7e.json (deflated 45%) 2025-12-04T12:13:24.3590286Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b44c71018f3811f.json (deflated 44%) 2025-12-04T12:13:24.3591807Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26728a48a74f0db0.json (deflated 37%) 2025-12-04T12:13:24.3593326Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9014cdddbced68f.json (deflated 37%) 2025-12-04T12:13:24.3594914Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-578fd43f9d90672b.json (deflated 37%) 2025-12-04T12:13:24.3596420Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-293b2ba538d5c1ee.json (deflated 37%) 2025-12-04T12:13:24.3597948Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea83e8b183ac592d.json (deflated 36%) 2025-12-04T12:13:24.3599501Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-436923729a977d14.json (deflated 37%) 2025-12-04T12:13:24.3601123Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5f688e692439013.json (deflated 37%) 2025-12-04T12:13:24.3602470Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4dc08961f281c036.json (deflated 39%) 2025-12-04T12:13:24.3603837Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da7051c1be9f2e2f.json (deflated 36%) 2025-12-04T12:13:24.3605189Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6675bf2f877791e.json (deflated 37%) 2025-12-04T12:13:24.3606540Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f31832653dbf03f.json (deflated 36%) 2025-12-04T12:13:24.3607900Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc72cfdb4feaf9b1.json (deflated 36%) 2025-12-04T12:13:24.3609252Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4abfc0edb4e297a.json (deflated 37%) 2025-12-04T12:13:24.3610612Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a979b5e962d58cfe.json (deflated 36%) 2025-12-04T12:13:24.3611961Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80387dc09d6e68db.json (deflated 38%) 2025-12-04T12:13:24.3613311Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec959ea151220e03.json (deflated 37%) 2025-12-04T12:13:24.3614656Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a47b279193b674dc.json (deflated 44%) 2025-12-04T12:13:24.3615985Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8aed26ab454245f.json (deflated 44%) 2025-12-04T12:13:24.3617330Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-498b958aacbfbe1c.json (deflated 35%) 2025-12-04T12:13:24.3618684Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ca585512d847c09.json (deflated 43%) 2025-12-04T12:13:24.3620024Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-654cb3e44738170f.json (deflated 43%) 2025-12-04T12:13:24.3621364Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28d3d63fc6efada0.json (deflated 37%) 2025-12-04T12:13:24.3622715Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00ace48f549d318a.json (deflated 43%) 2025-12-04T12:13:24.3624407Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-673dc73dcfb18eb0.json (deflated 45%) 2025-12-04T12:13:24.3625932Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfd6f3e2d018dce1.json (deflated 46%) 2025-12-04T12:13:24.3627626Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa80501c9f66d90c.json (deflated 36%) 2025-12-04T12:13:24.3629140Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e0dbc7fa11b6547.json (deflated 38%) 2025-12-04T12:13:24.3630697Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75e1b023ce49c02c.json (deflated 45%) 2025-12-04T12:13:24.3632213Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09a1c22c519254b6.json (deflated 44%) 2025-12-04T12:13:24.3633727Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c214051d59b82c2.json (deflated 38%) 2025-12-04T12:13:24.3635273Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed7ddfe1482b6f0b.json (deflated 44%) 2025-12-04T12:13:24.3636804Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a95a417b4eefa26.json (deflated 37%) 2025-12-04T12:13:24.3638328Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2ef4a32ca2b3dbf.json (deflated 37%) 2025-12-04T12:13:24.3639904Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5c9327604cb8f14.json (deflated 44%) 2025-12-04T12:13:24.3641256Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c38e0bbe495b143b.json (deflated 43%) 2025-12-04T12:13:24.3642600Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cb5b2f78eac5bfd.json (deflated 37%) 2025-12-04T12:13:24.3643954Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f44a2b1c1e232df9.json (deflated 37%) 2025-12-04T12:13:24.3645303Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7f8edc520bdf516e.json (deflated 37%) 2025-12-04T12:13:24.3646655Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-402e1eebb6506e23.json (deflated 38%) 2025-12-04T12:13:24.3647997Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d8076bdf0a69595.json (deflated 44%) 2025-12-04T12:13:24.3649339Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b468bab3124b13b3.json (deflated 37%) 2025-12-04T12:13:24.3650692Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e50d3219cb7bfbb6.json (deflated 44%) 2025-12-04T12:13:24.3652045Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53454fb25c79a584.json (deflated 44%) 2025-12-04T12:13:24.3653400Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ee8c69fe7ba0518.json (deflated 57%) 2025-12-04T12:13:24.3654738Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1220c9553e1d6abf.json (deflated 36%) 2025-12-04T12:13:24.3656092Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f13fc2d140babf7f.json (deflated 43%) 2025-12-04T12:13:24.3657619Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-557712f590042c69.json (deflated 43%) 2025-12-04T12:13:24.3659098Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce76f7d0c0c1d9a9.json (deflated 43%) 2025-12-04T12:13:24.3660513Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-323c51dd40cd5ff1.json (deflated 45%) 2025-12-04T12:13:24.3661941Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fbb3e53e76b1f52.json (deflated 38%) 2025-12-04T12:13:24.3663397Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17f25234f4eef787.json (deflated 37%) 2025-12-04T12:13:24.3665058Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-923db93bebc1ae0d.json (deflated 41%) 2025-12-04T12:13:24.3666974Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cab701c96fd8d8eb.json (deflated 41%) 2025-12-04T12:13:24.3669156Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-e799b1e80410ac4c.json (deflated 51%) 2025-12-04T12:13:24.3671169Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-17c54a2ebe2f0c44.json (deflated 52%) 2025-12-04T12:13:24.3673176Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-275ce76726773c8f.json (deflated 53%) 2025-12-04T12:13:24.3675176Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-086dc66f538a7679.json (deflated 53%) 2025-12-04T12:13:24.3676941Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51c0d6585102aef7.json (deflated 38%) 2025-12-04T12:13:24.3678434Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e2c29cc0f67c6e0.json (deflated 39%) 2025-12-04T12:13:24.3680036Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e04c95a8c2fac8bc.json (deflated 42%) 2025-12-04T12:13:24.3681384Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2143f4a3910ea5d.json (deflated 37%) 2025-12-04T12:13:24.3682718Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a623b45ea2222a4a.json (deflated 37%) 2025-12-04T12:13:24.3684048Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b489fc84ac56c90a.json (deflated 44%) 2025-12-04T12:13:24.3685391Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c606f93062a316ee.json (deflated 43%) 2025-12-04T12:13:24.3686722Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13409fa898261f4d.json (deflated 45%) 2025-12-04T12:13:24.3688055Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-822c6e3f32fc24f4.json (deflated 44%) 2025-12-04T12:13:24.3689390Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee05bfd466991d42.json (deflated 44%) 2025-12-04T12:13:24.3690717Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-997ca84c08e13a4f.json (deflated 37%) 2025-12-04T12:13:24.3692114Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1650b0a1cf5101b.json (deflated 37%) 2025-12-04T12:13:24.3693449Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f17cf2fe88278b7.json (deflated 37%) 2025-12-04T12:13:24.3694784Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d173d160413f56ff.json (deflated 37%) 2025-12-04T12:13:24.3696156Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c034757c66530030.json (deflated 49%) 2025-12-04T12:13:24.3697487Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19e760457ef6ff92.json (deflated 37%) 2025-12-04T12:13:24.3698817Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78ba10be7d074c23.json (deflated 37%) 2025-12-04T12:13:24.3700191Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b4bba108f9cb06d.json (deflated 37%) 2025-12-04T12:13:24.3701545Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-049ae9466a303b5c.json (deflated 47%) 2025-12-04T12:13:24.3702874Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44b7dc32fb33d263.json (deflated 37%) 2025-12-04T12:13:24.3704213Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75e1e407437545e3.json (deflated 37%) 2025-12-04T12:13:24.3705544Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70326d0de199e967.json (deflated 38%) 2025-12-04T12:13:24.3706890Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-537e26a62ece27c3.json (deflated 45%) 2025-12-04T12:13:24.3708517Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-276b8324a6920515.json (deflated 35%) 2025-12-04T12:13:24.3710014Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2263791b16c51a6.json (deflated 36%) 2025-12-04T12:13:24.3711527Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebaf98e72782bada.json (deflated 37%) 2025-12-04T12:13:24.3713037Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51f4be7c600c130e.json (deflated 37%) 2025-12-04T12:13:24.3714527Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64a85d7811c12838.json (deflated 37%) 2025-12-04T12:13:24.3716035Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-325d701d1891cfcd.json (deflated 57%) 2025-12-04T12:13:24.3717545Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bda5b12f5f6c4582.json (deflated 36%) 2025-12-04T12:13:24.3719046Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71b41420eeb91960.json (deflated 37%) 2025-12-04T12:13:24.3720615Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d12eb62cb70e222b.json (deflated 37%) 2025-12-04T12:13:24.3721944Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28e8d5b3cc52ed16.json (deflated 37%) 2025-12-04T12:13:24.3723341Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3575bf35b7b1c647.json (deflated 43%) 2025-12-04T12:13:24.3725088Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4522094cb3ebf4f.json (deflated 44%) 2025-12-04T12:13:24.3726601Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9492f0ab3ccaa218.json (deflated 36%) 2025-12-04T12:13:24.3728158Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4010fccffcf7594.json (deflated 36%) 2025-12-04T12:13:24.3729672Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ffb1b63fa92eef97.json (deflated 37%) 2025-12-04T12:13:24.3731190Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6439db28f2af0685.json (deflated 45%) 2025-12-04T12:13:24.3732746Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c2bb33ba5b4e7cc.json (deflated 44%) 2025-12-04T12:13:24.3734271Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95dc0d7d8e2b1b6d.json (deflated 42%) 2025-12-04T12:13:24.3735773Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cde022d82d2adcaa.json (deflated 42%) 2025-12-04T12:13:24.3737410Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3ea27d76bf44331.json (deflated 37%) 2025-12-04T12:13:24.3738751Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d08d23ef2c0e694a.json (deflated 38%) 2025-12-04T12:13:24.3740093Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-862e9783d4f59986.json (deflated 36%) 2025-12-04T12:13:24.3741413Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4662846f57df901f.json (deflated 37%) 2025-12-04T12:13:24.3742744Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-201ee2adc01d4dba.json (deflated 37%) 2025-12-04T12:13:24.3744082Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-232183083f1e6c18.json (deflated 39%) 2025-12-04T12:13:24.3745431Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92d5ea877ee6cee8.json (deflated 37%) 2025-12-04T12:13:24.3746770Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e7940eab8840d89.json (deflated 37%) 2025-12-04T12:13:24.3748385Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55d55fb078c2b684.json (deflated 36%) 2025-12-04T12:13:24.3749893Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a0928e0ccb8a6b10.json (deflated 36%) 2025-12-04T12:13:24.3751395Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2664d0ec37e344ff.json (deflated 43%) 2025-12-04T12:13:24.3752904Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32cc8afe70b3e7da.json (deflated 36%) 2025-12-04T12:13:24.3754406Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d3b40b7260221d.json (deflated 38%) 2025-12-04T12:13:24.3755913Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ac3d6524701cbf9.json (deflated 37%) 2025-12-04T12:13:24.3757504Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15963edc8117391b.json (deflated 37%) 2025-12-04T12:13:24.3759009Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc7ad1ec18a7bb1f.json (deflated 37%) 2025-12-04T12:13:24.3760671Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4b332b6754f948d.json (deflated 36%) 2025-12-04T12:13:24.3762026Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f36e4bfeb187dab.json (deflated 44%) 2025-12-04T12:13:24.3763366Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-046129b2924de1d6.json (deflated 37%) 2025-12-04T12:13:24.3764731Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1fb6eb7f4419f2c.json (deflated 45%) 2025-12-04T12:13:24.3766065Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb907762912a3d99.json (deflated 37%) 2025-12-04T12:13:24.3767388Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39f1cb056fff7a78.json (deflated 37%) 2025-12-04T12:13:24.3768732Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-285c65bf39f49db0.json (deflated 37%) 2025-12-04T12:13:24.3770069Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98105ea07b2795a4.json (deflated 36%) 2025-12-04T12:13:24.3771411Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecac8ed92cb8e9f2.json (deflated 37%) 2025-12-04T12:13:24.3772762Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abee6adb87748222.json (deflated 37%) 2025-12-04T12:13:24.3774089Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0c43579cac56c01.json (deflated 44%) 2025-12-04T12:13:24.3775421Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-838afcd07f98645e.json (deflated 44%) 2025-12-04T12:13:24.3776757Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-efef7c0c73e4a3a4.json (deflated 44%) 2025-12-04T12:13:24.3778091Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-279a5af71a085b78.json (deflated 45%) 2025-12-04T12:13:24.3779425Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03480c4b462e8e3c.json (deflated 45%) 2025-12-04T12:13:24.3780756Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32560068724b69d2.json (deflated 51%) 2025-12-04T12:13:24.3782092Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e0917bfdcbaebaa.json (deflated 44%) 2025-12-04T12:13:24.3783436Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-895373b2e146bc0e.json (deflated 37%) 2025-12-04T12:13:24.3784771Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c7263c0c2b97dd5.json (deflated 37%) 2025-12-04T12:13:24.3786091Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-607d80da638aa01c.json (deflated 37%) 2025-12-04T12:13:24.3787568Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-328e7d3fc5b20fc8.json (deflated 38%) 2025-12-04T12:13:24.3789240Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3fc9d94de1faf069.json (deflated 44%) 2025-12-04T12:13:24.3790751Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-725b83e279a2fc0a.json (deflated 37%) 2025-12-04T12:13:24.3792286Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd5d50730f4030d9.json (deflated 38%) 2025-12-04T12:13:24.3793804Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d65188606bc4fb38.json (deflated 37%) 2025-12-04T12:13:24.3795315Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-118492c3de10278a.json (deflated 57%) 2025-12-04T12:13:24.3796858Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-892e44a3547470c5.json (deflated 36%) 2025-12-04T12:13:24.3798343Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c889ae266192a3d2.json (deflated 36%) 2025-12-04T12:13:24.3799941Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91060a4136f305a0.json (deflated 44%) 2025-12-04T12:13:24.3801411Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abd0041fabca6f79.json (deflated 44%) 2025-12-04T12:13:24.3802752Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28c8471f94c139a5.json (deflated 37%) 2025-12-04T12:13:24.3804095Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-794deb0b1425e8fa.json (deflated 37%) 2025-12-04T12:13:24.3805419Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d2d9e77b4c2f3d8.json (deflated 37%) 2025-12-04T12:13:24.3806993Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-388ce163b11d3c23.json (deflated 41%) 2025-12-04T12:13:24.3808784Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-e6a8c3241228684f.json (deflated 41%) 2025-12-04T12:13:24.3810574Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cd4a02e6e5b27aca.json (deflated 51%) 2025-12-04T12:13:24.3812372Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-c2167be978f3dc6a.json (deflated 52%) 2025-12-04T12:13:24.3814142Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-14585f3cbdd695ec.json (deflated 53%) 2025-12-04T12:13:24.3815929Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-bfadba8f7e8ab3cd.json (deflated 53%) 2025-12-04T12:13:24.3817503Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24ebf91103f6fea1.json (deflated 38%) 2025-12-04T12:13:24.3818863Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-206af608512f699e.json (deflated 38%) 2025-12-04T12:13:24.3820278Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f82acc534672cdf.json (deflated 42%) 2025-12-04T12:13:24.3821619Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bc0e514e7eb9947.json (deflated 37%) 2025-12-04T12:13:24.3822966Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5301f1689efddffd.json (deflated 37%) 2025-12-04T12:13:24.3824825Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39f8f74b2c4bb356.json (deflated 44%) 2025-12-04T12:13:24.3826343Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4deb287bf0f15502.json (deflated 43%) 2025-12-04T12:13:24.3827926Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-69b712ded204926d.json (deflated 45%) 2025-12-04T12:13:24.3829738Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac8d4732ea94303d.json (deflated 44%) 2025-12-04T12:13:24.3831261Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b15b1dbca96c31dd.json (deflated 44%) 2025-12-04T12:13:24.3832773Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5235361330778453.json (deflated 37%) 2025-12-04T12:13:24.3834278Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d366d5954251fcda.json (deflated 37%) 2025-12-04T12:13:24.3835796Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b47d5bdbdea81123.json (deflated 37%) 2025-12-04T12:13:24.3837320Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a24f6ee3b330fe2.json (deflated 37%) 2025-12-04T12:13:24.3838842Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6ba4b460a331f8f.json (deflated 49%) 2025-12-04T12:13:24.3840464Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-676fe7068d5b8cce.json (deflated 37%) 2025-12-04T12:13:24.3841816Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a360a85ac03dda59.json (deflated 37%) 2025-12-04T12:13:24.3843166Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1708cca9dc0e798.json (deflated 37%) 2025-12-04T12:13:24.3844511Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a8ced54e15b6002.json (deflated 47%) 2025-12-04T12:13:24.3845864Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c9833d98fcaa164.json (deflated 37%) 2025-12-04T12:13:24.3847202Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eec63fa6ac957019.json (deflated 37%) 2025-12-04T12:13:24.3848544Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9032fc63d800720d.json (deflated 38%) 2025-12-04T12:13:24.3849893Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9237cfd956a6ea6.json (deflated 45%) 2025-12-04T12:13:24.3851235Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d3dac1ff57d770a.json (deflated 36%) 2025-12-04T12:13:24.3852659Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e71aa694c478fc1e.json (deflated 37%) 2025-12-04T12:13:24.3854013Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3303d981481b9d1f.json (deflated 37%) 2025-12-04T12:13:24.3855364Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10e39578cdaa15c7.json (deflated 37%) 2025-12-04T12:13:24.3856944Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1b45f0468676a59.json (deflated 37%) 2025-12-04T12:13:24.3858373Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e83f111f9b87d25.json (deflated 57%) 2025-12-04T12:13:24.3859795Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10563eb7a07575a6.json (deflated 36%) 2025-12-04T12:13:24.3861253Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc37255b8c6ba83b.json (deflated 37%) 2025-12-04T12:13:24.3862685Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13cdaac161151c73.json (deflated 37%) 2025-12-04T12:13:24.3864291Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6fc399e73bd2cfea.json (deflated 37%) 2025-12-04T12:13:24.3865774Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c09d245ecad9319.json (deflated 43%) 2025-12-04T12:13:24.3867238Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0e3e1dcb90d6929.json (deflated 44%) 2025-12-04T12:13:24.3868991Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51c71b398d03a4df.json (deflated 36%) 2025-12-04T12:13:24.3870523Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3006a46890f44c5.json (deflated 36%) 2025-12-04T12:13:24.3872044Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-267dee8bb7025f13.json (deflated 37%) 2025-12-04T12:13:24.3873555Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a7645ce5728017b.json (deflated 45%) 2025-12-04T12:13:24.3875080Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-deb9bfbb4bd80c4b.json (deflated 44%) 2025-12-04T12:13:24.3876617Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e11a11c30a005f3.json (deflated 42%) 2025-12-04T12:13:24.3878150Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c85cce560acdb42f.json (deflated 42%) 2025-12-04T12:13:24.3879784Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96d41dfdc3fe6dba.json (deflated 37%) 2025-12-04T12:13:24.3881271Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c26119bec135fb9d.json (deflated 37%) 2025-12-04T12:13:24.3882623Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-114850235e770dcb.json (deflated 36%) 2025-12-04T12:13:24.3883973Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ed8dfc73ef01d6d.json (deflated 37%) 2025-12-04T12:13:24.3885321Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49b473d112cc4546.json (deflated 37%) 2025-12-04T12:13:24.3886716Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc13b9aff20c070d.json (deflated 39%) 2025-12-04T12:13:24.3888070Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04e707708d96db67.json (deflated 37%) 2025-12-04T12:13:24.3889409Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4697a0ab5cd20b70.json (deflated 37%) 2025-12-04T12:13:24.3890787Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a417a16a93526fa4.json (deflated 36%) 2025-12-04T12:13:24.3892136Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ddfeed99e9cdec6.json (deflated 36%) 2025-12-04T12:13:24.3893509Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16b81ab28b107f75.json (deflated 43%) 2025-12-04T12:13:24.3894853Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cff05e49a4594759.json (deflated 36%) 2025-12-04T12:13:24.3896207Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db1fdc0a1e7f6276.json (deflated 38%) 2025-12-04T12:13:24.3897567Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cb2bb364c0ae81e.json (deflated 37%) 2025-12-04T12:13:24.3898905Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16f5e50845c16298.json (deflated 37%) 2025-12-04T12:13:24.3900247Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d791b90d6362f3c0.json (deflated 37%) 2025-12-04T12:13:24.3901608Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96506a809a7d9377.json (deflated 36%) 2025-12-04T12:13:24.3902952Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-140f4b27a4309298.json (deflated 44%) 2025-12-04T12:13:24.3904307Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-541ecd361c5cbd66.json (deflated 37%) 2025-12-04T12:13:24.3905642Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0707caf97ad64431.json (deflated 45%) 2025-12-04T12:13:24.3906992Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a41a3afe18e29798.json (deflated 37%) 2025-12-04T12:13:24.3908657Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-384c7454c26fa694.json (deflated 37%) 2025-12-04T12:13:24.3910168Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39a728e06646a1d3.json (deflated 37%) 2025-12-04T12:13:24.3911675Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ddaf75327e52f1c1.json (deflated 36%) 2025-12-04T12:13:24.3913197Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ec25664c586fadf.json (deflated 38%) 2025-12-04T12:13:24.3914718Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08df3d74d553659c.json (deflated 37%) 2025-12-04T12:13:24.3916237Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d31b0daadd5824cb.json (deflated 44%) 2025-12-04T12:13:24.3917827Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63eea3de02c3a6ef.json (deflated 44%) 2025-12-04T12:13:24.3919339Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-656d9998e6c222a4.json (deflated 44%) 2025-12-04T12:13:24.3920977Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6acaa860eb966f60.json (deflated 45%) 2025-12-04T12:13:24.3922358Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce80904a0746c1f8.json (deflated 45%) 2025-12-04T12:13:24.3923829Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13fade6629382363.json (deflated 51%) 2025-12-04T12:13:24.3925533Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c48a0039a0795ab.json (deflated 44%) 2025-12-04T12:13:24.3927118Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1960e6db1730a4a3.json (deflated 37%) 2025-12-04T12:13:24.3928643Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b68267f25edfed92.json (deflated 37%) 2025-12-04T12:13:24.3930164Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-011a1d9cca18ee04.json (deflated 37%) 2025-12-04T12:13:24.3931689Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e9ae608f7e37be8.json (deflated 38%) 2025-12-04T12:13:24.3933189Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07300589f39fc3ac.json (deflated 44%) 2025-12-04T12:13:24.3934713Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cbc9db131dce2e6.json (deflated 37%) 2025-12-04T12:13:24.3936241Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-458e56e213b7ad87.json (deflated 37%) 2025-12-04T12:13:24.3937839Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d87679cc1c529e2d.json (deflated 37%) 2025-12-04T12:13:24.3939175Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4b08dbbb6915206.json (deflated 57%) 2025-12-04T12:13:24.3940523Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30aed6037f140bb7.json (deflated 36%) 2025-12-04T12:13:24.3941863Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-689b3f622d74137a.json (deflated 36%) 2025-12-04T12:13:24.3943223Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cb150e7fa6adefe.json (deflated 44%) 2025-12-04T12:13:24.3944578Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-773bb593cee0da12.json (deflated 44%) 2025-12-04T12:13:24.3945916Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3c6b8d3271f370c.json (deflated 37%) 2025-12-04T12:13:24.3947318Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7116fa9f879b5f5c.json (deflated 37%) 2025-12-04T12:13:24.3948966Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab516b6bd9172a5a.json (deflated 37%) 2025-12-04T12:13:24.3971668Z ##[group]Run # Remove any previous test reports if they exist 2025-12-04T12:13:24.3972324Z # Remove any previous test reports if they exist 2025-12-04T12:13:24.3972752Z rm -f test-reports-*.zip 2025-12-04T12:13:24.3973268Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-12-04T12:13:24.3979569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:24.3979982Z env: 2025-12-04T12:13:24.3980276Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:24.3980696Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:24.3981030Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:24.3981606Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:24.3982302Z FILE_SUFFIX: test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912 2025-12-04T12:13:24.3982800Z ##[endgroup] 2025-12-04T12:13:24.4142746Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-98b5e89a175f7fbc.xml (deflated 86%) 2025-12-04T12:13:24.4144375Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-bb030510459950ee.xml (deflated 77%) 2025-12-04T12:13:24.4145766Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-0f19060d211e471b.xml (deflated 90%) 2025-12-04T12:13:24.4147153Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-4e3b8ac35e315f80.xml (deflated 77%) 2025-12-04T12:13:24.4148841Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_uneven/distributed.fsdp.test_fsdp_uneven-7270531c264bcf9a.xml (deflated 28%) 2025-12-04T12:13:24.4150262Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-ed7a8589598da4ef.xml (deflated 77%) 2025-12-04T12:13:24.4151670Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-68059153dd3ae49c.xml (deflated 77%) 2025-12-04T12:13:24.4153093Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-dd8d334d3533ae96.xml (deflated 77%) 2025-12-04T12:13:24.4154498Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-571ae876e059f56e.xml (deflated 77%) 2025-12-04T12:13:24.4155902Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-6f9ee7a291766819.xml (deflated 77%) 2025-12-04T12:13:24.4157308Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-7369439069792ace.xml (deflated 77%) 2025-12-04T12:13:24.4158715Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_input/distributed.fsdp.test_fsdp_input-f5da965ca571529c.xml (deflated 28%) 2025-12-04T12:13:24.4160260Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-551e66d027b33cdd.xml (deflated 77%) 2025-12-04T12:13:24.4161683Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-8583a5ccabf54360.xml (deflated 77%) 2025-12-04T12:13:24.4163132Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-58c9173ffcd43d6d.xml (deflated 77%) 2025-12-04T12:13:24.4164575Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_traversal/distributed.fsdp.test_fsdp_traversal-3a68936c644e7b41.xml (deflated 28%) 2025-12-04T12:13:24.4166029Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-88a9f26e5fb62840.xml (deflated 85%) 2025-12-04T12:13:24.4167503Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-52aebc2b2ad7446f.xml (deflated 78%) 2025-12-04T12:13:24.4169095Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-c4e36de52e6d69c2.xml (deflated 78%) 2025-12-04T12:13:24.4170574Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_checkpoint/distributed.fsdp.test_fsdp_checkpoint-448a44a24c52f556.xml (deflated 28%) 2025-12-04T12:13:24.4171980Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a25290001c1b014e.xml (deflated 78%) 2025-12-04T12:13:24.4173377Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-cc80cfecc5abaf73.xml (deflated 78%) 2025-12-04T12:13:24.4174716Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-00aad17389ddcea3.xml (deflated 78%) 2025-12-04T12:13:24.4176063Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc7285fa7f4227.xml (deflated 78%) 2025-12-04T12:13:24.4177446Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-69d9e5e2bc4a3a7d.xml (deflated 86%) 2025-12-04T12:13:24.4178795Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c1c17128b5541633.xml (deflated 78%) 2025-12-04T12:13:24.4180131Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-156a51594e715e30.xml (deflated 78%) 2025-12-04T12:13:24.4181451Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2e6a295727102ba7.xml (deflated 78%) 2025-12-04T12:13:24.4182790Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-2bb3299d57ce6c2b.xml (deflated 78%) 2025-12-04T12:13:24.4184136Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-b434b54a558efa89.xml (deflated 78%) 2025-12-04T12:13:24.4185491Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-bd75c8cd2cf8ff9e.xml (deflated 78%) 2025-12-04T12:13:24.4186832Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f7fa0727268f1a56.xml (deflated 78%) 2025-12-04T12:13:24.4188435Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f90badc543da9251.xml (deflated 87%) 2025-12-04T12:13:24.4189825Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-31be29224184e218.xml (deflated 78%) 2025-12-04T12:13:24.4191209Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1c567b53e42c8343.xml (deflated 78%) 2025-12-04T12:13:24.4192574Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-3db6e54b58890527.xml (deflated 78%) 2025-12-04T12:13:24.4193958Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-5f8a5af0d826239e.xml (deflated 90%) 2025-12-04T12:13:24.4195341Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-1a3a2c29e874821d.xml (deflated 86%) 2025-12-04T12:13:24.4196718Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-097bd188165cd189.xml (deflated 90%) 2025-12-04T12:13:24.4198111Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-efc9fa1f0bd357d6.xml (deflated 78%) 2025-12-04T12:13:24.4199480Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-648e0de30361d669.xml (deflated 78%) 2025-12-04T12:13:24.4200934Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-21aecc6f8d181c25.xml (deflated 78%) 2025-12-04T12:13:24.4202377Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-ab70b17c1ec5b8c5.xml (deflated 86%) 2025-12-04T12:13:24.4203722Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e033904d3fffd97a.xml (deflated 78%) 2025-12-04T12:13:24.4205051Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-f030d3020f190621.xml (deflated 77%) 2025-12-04T12:13:24.4206423Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-da8aed1c14f5aeeb.xml (deflated 77%) 2025-12-04T12:13:24.4207776Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-c91f300ede57a5d9.xml (deflated 77%) 2025-12-04T12:13:24.4209136Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-a2a9be50a03cddfc.xml (deflated 77%) 2025-12-04T12:13:24.4210502Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-be7435212e20eea8.xml (deflated 77%) 2025-12-04T12:13:24.4211848Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-20cc23628278ddca.xml (deflated 77%) 2025-12-04T12:13:24.4213188Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-e7d51fdd1a2eae54.xml (deflated 28%) 2025-12-04T12:13:24.4214686Z adding: test/test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-19176bf0e30ddda9.xml (deflated 76%) 2025-12-04T12:13:24.4216150Z adding: test/test-reports/python-pytest/distributed.test_c10d_ops_nccl/distributed.test_c10d_ops_nccl-a72c588f9917f825.xml (deflated 86%) 2025-12-04T12:13:24.4217648Z adding: test/test-reports/python-pytest/distributed.elastic.multiprocessing.api_test/distributed.elastic.multiprocessing.api_test-3223bbc58a6df58f.xml (deflated 82%) 2025-12-04T12:13:24.4219255Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-c6dfa06eb71f87cd.xml (deflated 87%) 2025-12-04T12:13:24.4220746Z adding: test/test-reports/python-pytest/distributed.tensor.test_dtensor_export/distributed.tensor.test_dtensor_export-8b65af0c5c32155a.xml (deflated 69%) 2025-12-04T12:13:24.4222259Z adding: test/test-reports/python-pytest/distributed._tools.test_fake_collectives/distributed._tools.test_fake_collectives-619983be33386a60.xml (deflated 38%) 2025-12-04T12:13:24.4224133Z adding: test/test-reports/python-pytest/distributed.test_control_collectives/distributed.test_control_collectives-3f2bbee5075aa12d.xml (deflated 84%) 2025-12-04T12:13:24.4225624Z adding: test/test-reports/python-pytest/distributed.test_collective_utils/distributed.test_collective_utils-9394cde5cb4e06c2.xml (deflated 78%) 2025-12-04T12:13:24.4227136Z adding: test/test-reports/python-pytest/distributed.test_c10d_object_collectives/distributed.test_c10d_object_collectives-d97a03c8ce93a326.xml (deflated 80%) 2025-12-04T12:13:24.4228813Z adding: test/test-reports/python-pytest/distributed.tensor.debug.test_op_coverage/distributed.tensor.debug.test_op_coverage-ebb808e01daa256c.xml (deflated 35%) 2025-12-04T12:13:24.4230510Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-3fe654a75937f34f.xml (deflated 91%) 2025-12-04T12:13:24.4232136Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-2a8ca78a6e78af3b.xml (deflated 65%) 2025-12-04T12:13:24.4233691Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-8d04d60499ed13c6.xml (deflated 37%) 2025-12-04T12:13:24.4235431Z adding: test/test-reports/python-pytest/distributed.tensor.test_dtensor_compile/distributed.tensor.test_dtensor_compile-b72028e76a0d9d1d.xml (deflated 86%) 2025-12-04T12:13:24.4237050Z adding: test/test-reports/python-pytest/distributed.test_aten_comm_compute_reordering/distributed.test_aten_comm_compute_reordering-7d1c62987df9c762.xml (deflated 83%) 2025-12-04T12:13:24.4238629Z adding: test/test-reports/python-pytest/distributed._tools.test_sac_estimator/distributed._tools.test_sac_estimator-b0bd4af0f86e3ece.xml (deflated 51%) 2025-12-04T12:13:24.4240252Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-c7b456a4f8311d2d.xml (deflated 63%) 2025-12-04T12:13:24.4241851Z adding: test/test-reports/python-pytest/distributed.checkpoint._experimental.test_builder/distributed.checkpoint._experimental.test_builder-63eb721b9e0f0e0e.xml (deflated 69%) 2025-12-04T12:13:24.4243582Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate_with_fsdp/distributed._composable.test_replicate_with_fsdp-4c94f165a9954ceb.xml (deflated 70%) 2025-12-04T12:13:24.4245260Z adding: test/test-reports/python-pytest/distributed.tensor.test_xla_integration/distributed.tensor.test_xla_integration-953545bf454723f1.xml (deflated 74%) 2025-12-04T12:13:24.4246856Z adding: test/test-reports/python-pytest/distributed.checkpoint._experimental.test_types/distributed.checkpoint._experimental.test_types-02c661f341e1f68e.xml (deflated 60%) 2025-12-04T12:13:24.4248636Z adding: test/test-reports/python-pytest/distributed.tensor.experimental.test_register_sharding/distributed.tensor.experimental.test_register_sharding-ee24555d8c546070.xml (deflated 62%) 2025-12-04T12:13:24.4250196Z adding: test/test-reports/python-pytest/distributed.test_backends/distributed.test_backends-11b04c9998663597.xml (deflated 49%) 2025-12-04T12:13:24.4251585Z adding: test/test-reports/python-pytest/distributed.tensor.test_experimental_ops/distributed.tensor.test_experimental_ops-1d5fbe2b841d7120.xml (deflated 68%) 2025-12-04T12:13:24.4253227Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_quantized_hf_storage/distributed.checkpoint.test_quantized_hf_storage-b35612bb520e65c5.xml (deflated 52%) 2025-12-04T12:13:24.4255097Z adding: test/test-reports/python-pytest/distributed._composable.test_composability.test_pp_composability/distributed._composable.test_composability.test_pp_composability-6aa49750a7598206.xml (deflated 95%) 2025-12-04T12:13:24.4256986Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_async_process_executor/distributed.checkpoint.test_async_process_executor-257bf63a3e72d6aa.xml (deflated 70%) 2025-12-04T12:13:24.4258583Z adding: test/test-reports/python-pytest/distributed.tensor.test_tensor_ops/distributed.tensor.test_tensor_ops-dd61e7522cc11d1f.xml (deflated 91%) 2025-12-04T12:13:24.4259933Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-2d037d076dad7e9e.xml (deflated 88%) 2025-12-04T12:13:24.4261541Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_ignore_params/distributed._composable.fsdp.test_fully_shard_ignore_params-3bd5835b23ec8132.xml (deflated 35%) 2025-12-04T12:13:24.4263347Z adding: test/test-reports/python-pytest/distributed.checkpoint._experimental.test_staging/distributed.checkpoint._experimental.test_staging-07a6a2de828c3109.xml (deflated 72%) 2025-12-04T12:13:24.4265157Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion-7932544576c43c55.xml (deflated 39%) 2025-12-04T12:13:24.4266838Z adding: test/test-reports/python-pytest/distributed.tensor.test_embedding_ops/distributed.tensor.test_embedding_ops-389c4439b22d140b.xml (deflated 72%) 2025-12-04T12:13:24.4268683Z adding: test/test-reports/python-pytest/distributed.tensor.experimental.test_local_map/distributed.tensor.experimental.test_local_map-43637351f823edb4.xml (deflated 74%) 2025-12-04T12:13:24.4270296Z adding: test/test-reports/python-pytest/distributed.test_local_tensor/distributed.test_local_tensor-2d16d75957767ef5.xml (deflated 81%) 2025-12-04T12:13:24.4271855Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-f6baa7b934e7caab.xml (deflated 73%) 2025-12-04T12:13:24.4273558Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_tp_checkpoint/distributed.checkpoint.test_tp_checkpoint-11d7a0d46082f98d.xml (deflated 54%) 2025-12-04T12:13:24.4275144Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-4e9f4b21d1bcf8f6.xml (deflated 73%) 2025-12-04T12:13:24.4276765Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_random_state/distributed.tensor.parallel.test_tp_random_state-3d1a164d85d827e3.xml (deflated 37%) 2025-12-04T12:13:24.4278401Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_planner/distributed.checkpoint.test_planner-6c9d97ee371ac2d1.xml (deflated 77%) 2025-12-04T12:13:24.4280163Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_checkpoint/distributed.checkpoint.test_dtensor_checkpoint-cb541eaa40da104f.xml (deflated 39%) 2025-12-04T12:13:24.4281771Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule/distributed.pipelining.test_schedule-87ecb28af2fb8a27.xml (deflated 89%) 2025-12-04T12:13:24.4283421Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-b60f2694fe7d2c52.xml (deflated 62%) 2025-12-04T12:13:24.4284873Z adding: test/test-reports/python-pytest/distributed.test_run/distributed.test_run-ff2c644d612e7850.xml (deflated 62%) 2025-12-04T12:13:24.4286110Z adding: test/test-reports/python-pytest/distributed.tensor.test_math_ops/distributed.tensor.test_math_ops-f7126468d0203b5f.xml (deflated 88%) 2025-12-04T12:13:24.4287443Z adding: test/test-reports/python-pytest/distributed.fsdp.test_utils/distributed.fsdp.test_utils-62adb141232f9dae.xml (deflated 69%) 2025-12-04T12:13:24.4288896Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-9094c50849cf3609.xml (deflated 86%) 2025-12-04T12:13:24.4290592Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_compile/distributed._composable.fsdp.test_fully_shard_compile-0fa50fae7281ecc5.xml (deflated 87%) 2025-12-04T12:13:24.4292356Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-db4a02d440fa1c43.xml (deflated 61%) 2025-12-04T12:13:24.4293995Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_pg_transport/distributed.checkpoint.test_pg_transport-f8a475bbd4225371.xml (deflated 78%) 2025-12-04T12:13:24.4295745Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-5f953c8bff259762.xml (deflated 79%) 2025-12-04T12:13:24.4297428Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_utils/distributed.checkpoint.test_utils-3867a40d83a2b51a.xml (deflated 80%) 2025-12-04T12:13:24.4299138Z adding: test/test-reports/python-pytest/distributed.checkpoint._experimental.test_checkpoint_process/distributed.checkpoint._experimental.test_checkpoint_process-8b0b149c7b33e94c.xml (deflated 83%) 2025-12-04T12:13:24.4300817Z adding: test/test-reports/python-pytest/distributed.tensor.test_view_ops/distributed.tensor.test_view_ops-04bcb1d8735e8634.xml (deflated 82%) 2025-12-04T12:13:24.4302211Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-0e669e7d48e3457f.xml (deflated 95%) 2025-12-04T12:13:24.4303704Z adding: test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-e79a881124a4ef8b.xml (deflated 75%) 2025-12-04T12:13:24.4305407Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint/distributed.checkpoint.test_file_system_checkpoint-8e7248bfdb62ae56.xml (deflated 80%) 2025-12-04T12:13:24.4307193Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_file_system_checkpoint_cpu/distributed.checkpoint.test_file_system_checkpoint_cpu-86828f53aae09692.xml (deflated 81%) 2025-12-04T12:13:24.4309111Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm_hooks/distributed.fsdp.test_fsdp_comm_hooks-e30f0f398ca36b1a.xml (deflated 91%) 2025-12-04T12:13:24.4310546Z adding: test/test-reports/python-pytest/distributed._shard.test_sharder/distributed._shard.test_sharder-c376df3131abbe36.xml (deflated 53%) 2025-12-04T12:13:24.4312132Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_tensor_ops/distributed._shard.sharded_tensor.ops.test_tensor_ops-130017345592955f.xml (deflated 71%) 2025-12-04T12:13:24.4313854Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-4367c9892b5119af.xml (deflated 60%) 2025-12-04T12:13:24.4315281Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-57a88f260c958571.xml (deflated 88%) 2025-12-04T12:13:24.4316559Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5c2225b883027b57.xml (deflated 34%) 2025-12-04T12:13:24.4317869Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e989bf90bb499159.xml (deflated 33%) 2025-12-04T12:13:24.4319171Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5358d199d91e916e.xml (deflated 34%) 2025-12-04T12:13:24.4320555Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e8f01c1258f32697.xml (deflated 33%) 2025-12-04T12:13:24.4321813Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-ac92f4ba57cf76c4.xml (deflated 33%) 2025-12-04T12:13:24.4323086Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-eda4e22775599b5a.xml (deflated 33%) 2025-12-04T12:13:24.4324684Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-f2042aa2d074e648.xml (deflated 34%) 2025-12-04T12:13:24.4325981Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-eace822c5c4d90b0.xml (deflated 34%) 2025-12-04T12:13:24.4327278Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e2aaf0e6db1856f6.xml (deflated 35%) 2025-12-04T12:13:24.4328579Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-aba1e98d5f917e6f.xml (deflated 35%) 2025-12-04T12:13:24.4329887Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-c8752499cbdebb6b.xml (deflated 35%) 2025-12-04T12:13:24.4331186Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-e365ef3c3de4e526.xml (deflated 35%) 2025-12-04T12:13:24.4332467Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-3d0c386b38968978.xml (deflated 34%) 2025-12-04T12:13:24.4333769Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-c9893cb7ff9eb7cb.xml (deflated 35%) 2025-12-04T12:13:24.4335137Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-d8ca55d72dd5d963.xml (deflated 35%) 2025-12-04T12:13:24.4336691Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-d6cc878cd0b0d21e.xml (deflated 35%) 2025-12-04T12:13:24.4338284Z adding: test/test-reports/python-pytest/distributed.test_pg_wrapper/distributed.test_pg_wrapper-5399ee4967a63bf6.xml (deflated 35%) 2025-12-04T12:13:24.4339837Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_forward/distributed.fsdp.test_fsdp_multiple_forward-97a90e2322241ed5.xml (deflated 28%) 2025-12-04T12:13:24.4341605Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_binary_cmp/distributed._shard.sharded_tensor.ops.test_binary_cmp-1b60820ad9df53ea.xml (deflated 69%) 2025-12-04T12:13:24.4343351Z adding: test/test-reports/python-pytest/distributed.nn.jit.test_instantiator/distributed.nn.jit.test_instantiator-26bb23415f1ce435.xml (deflated 61%) 2025-12-04T12:13:24.4345058Z adding: test/test-reports/python-pytest/distributed._shard.sharding_spec.test_sharding_spec/distributed._shard.sharding_spec.test_sharding_spec-b7677c73b05bb97d.xml (deflated 81%) 2025-12-04T12:13:24.4346619Z adding: test/test-reports/python-pytest/distributed.test_nccl/distributed.test_nccl-6ab9552fe0bfe15f.xml (deflated 77%) 2025-12-04T12:13:24.4348349Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_misc/distributed.fsdp.test_fsdp_misc-ff29f8b606fc4847.xml (deflated 79%) 2025-12-04T12:13:24.4349852Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_meta/distributed.fsdp.test_fsdp_meta-8050e6a6d02fe535.xml (deflated 88%) 2025-12-04T12:13:24.4351296Z adding: test/test-reports/python-pytest/distributed.test_data_parallel/distributed.test_data_parallel-c1f5c0fcbf415006.xml (deflated 81%) 2025-12-04T12:13:24.4352904Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e8d73f4591d62db9.xml (deflated 35%) 2025-12-04T12:13:24.4370858Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9a221c145b1a74c2.xml (deflated 35%) 2025-12-04T12:13:24.4372467Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-08cb4ec451a35d57.xml (deflated 36%) 2025-12-04T12:13:24.4374036Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-dee1ac91d0c754d6.xml (deflated 36%) 2025-12-04T12:13:24.4375576Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-350598816b45384d.xml (deflated 38%) 2025-12-04T12:13:24.4377128Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-510c0213c3d43ce4.xml (deflated 38%) 2025-12-04T12:13:24.4378818Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b9f09d55835d87af.xml (deflated 38%) 2025-12-04T12:13:24.4380363Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ed820609834e1378.xml (deflated 37%) 2025-12-04T12:13:24.4381920Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c88e834a8b4dbb20.xml (deflated 36%) 2025-12-04T12:13:24.4383466Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-f0b18dfb3a59dad1.xml (deflated 37%) 2025-12-04T12:13:24.4385020Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4743d357d297f6dc.xml (deflated 37%) 2025-12-04T12:13:24.4386570Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4662c39cb6a4b928.xml (deflated 37%) 2025-12-04T12:13:24.4388453Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-73028dd94dff8c49.xml (deflated 36%) 2025-12-04T12:13:24.4390176Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-3ee7cb942a2cc6e3.xml (deflated 36%) 2025-12-04T12:13:24.4391777Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-5669ef320dd02577.xml (deflated 37%) 2025-12-04T12:13:24.4393391Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c3e4151039d68fa8.xml (deflated 37%) 2025-12-04T12:13:24.4395046Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-024c7a32a248042e.xml (deflated 36%) 2025-12-04T12:13:24.4396643Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c97e92ee7b028ae3.xml (deflated 36%) 2025-12-04T12:13:24.4398281Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d8dd34d3359043f5.xml (deflated 36%) 2025-12-04T12:13:24.4399986Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-934a5e5071fe9c36.xml (deflated 36%) 2025-12-04T12:13:24.4401544Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d42c1866d87ab3db.xml (deflated 36%) 2025-12-04T12:13:24.4403095Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d46cf2452bc01536.xml (deflated 36%) 2025-12-04T12:13:24.4404635Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8f4f39fa980e82f2.xml (deflated 36%) 2025-12-04T12:13:24.4406189Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0652764081d6da21.xml (deflated 36%) 2025-12-04T12:13:24.4407738Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-610c219c10381302.xml (deflated 36%) 2025-12-04T12:13:24.4409289Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ff765bddb1f8ea71.xml (deflated 37%) 2025-12-04T12:13:24.4410849Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9be5a3fd2574b057.xml (deflated 37%) 2025-12-04T12:13:24.4412395Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-df5bf940ae0a6258.xml (deflated 36%) 2025-12-04T12:13:24.4413957Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a70e67bbc896acc2.xml (deflated 36%) 2025-12-04T12:13:24.4415518Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e5a6bcea490b9fda.xml (deflated 36%) 2025-12-04T12:13:24.4417079Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-1d02ec45c02f60fe.xml (deflated 36%) 2025-12-04T12:13:24.4418631Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-589f53701a6c7b6a.xml (deflated 36%) 2025-12-04T12:13:24.4420192Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-37a88acccbf1ecd0.xml (deflated 36%) 2025-12-04T12:13:24.4421760Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-9e0e61cd971b69bb.xml (deflated 36%) 2025-12-04T12:13:24.4423369Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-a25ed88c721098f2.xml (deflated 36%) 2025-12-04T12:13:24.4425313Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c9d57215b8ca5d9f.xml (deflated 36%) 2025-12-04T12:13:24.4426910Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-07a2c7293983a21f.xml (deflated 36%) 2025-12-04T12:13:24.4428678Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-e7d26bc200bae427.xml (deflated 36%) 2025-12-04T12:13:24.4430286Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-c4303c90119175e3.xml (deflated 35%) 2025-12-04T12:13:24.4431883Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-b08f37975720f6b4.xml (deflated 36%) 2025-12-04T12:13:24.4433533Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-75b604dc9c04c4c7.xml (deflated 35%) 2025-12-04T12:13:24.4435131Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-8dc1d7aea93bdb34.xml (deflated 36%) 2025-12-04T12:13:24.4436738Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4a3867d6b6932f10.xml (deflated 36%) 2025-12-04T12:13:24.4438339Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0318829c01be08fa.xml (deflated 36%) 2025-12-04T12:13:24.4440029Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-693b8cadac47d9c2.xml (deflated 35%) 2025-12-04T12:13:24.4441586Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-0ecc816e8885b9bc.xml (deflated 35%) 2025-12-04T12:13:24.4443141Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d3419121e211fb91.xml (deflated 36%) 2025-12-04T12:13:24.4444690Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-2c51cc1e26d5acf3.xml (deflated 35%) 2025-12-04T12:13:24.4446245Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-4746fedc7b470508.xml (deflated 36%) 2025-12-04T12:13:24.4447795Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-d82b54717ce755b5.xml (deflated 37%) 2025-12-04T12:13:24.4449351Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-730ab8cf8f5003b5.xml (deflated 37%) 2025-12-04T12:13:24.4450900Z adding: test/test-reports/python-pytest/distributed.rpc.cuda.test_tensorpipe_agent/distributed.rpc.cuda.test_tensorpipe_agent-ef621cb706a80f36.xml (deflated 35%) 2025-12-04T12:13:24.4452454Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_unshard_params/distributed.fsdp.test_fsdp_unshard_params-49f6028e4feadb15.xml (deflated 83%) 2025-12-04T12:13:24.4454047Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_init/distributed._shard.sharded_tensor.ops.test_init-b148a45a231b8de9.xml (deflated 64%) 2025-12-04T12:13:24.4455778Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.ops.test_embedding_bag/distributed._shard.sharded_tensor.ops.test_embedding_bag-0fb42e6113dec695.xml (deflated 56%) 2025-12-04T12:13:24.4457423Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e3270986f520c455.xml (deflated 36%) 2025-12-04T12:13:24.4458771Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9f07ad58b8cd3f14.xml (deflated 35%) 2025-12-04T12:13:24.4460109Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-38ae8a6c9446176c.xml (deflated 35%) 2025-12-04T12:13:24.4461475Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-9be7dc10fc456dd4.xml (deflated 35%) 2025-12-04T12:13:24.4462802Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a4bb8d2a63c3e77a.xml (deflated 35%) 2025-12-04T12:13:24.4464137Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a65cb5796fd83f18.xml (deflated 35%) 2025-12-04T12:13:24.4465508Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-44d720723f83b050.xml (deflated 35%) 2025-12-04T12:13:24.4466841Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-a0db0c7d2253e897.xml (deflated 36%) 2025-12-04T12:13:24.4468614Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-4d41a04cacb8591f.xml (deflated 35%) 2025-12-04T12:13:24.4470012Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-79c50cf35aedde1f.xml (deflated 35%) 2025-12-04T12:13:24.4471383Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-e36163ffaa730c12.xml (deflated 35%) 2025-12-04T12:13:24.4472852Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_use_orig_params/distributed.fsdp.test_fsdp_use_orig_params-c1df23351bfa5ee8.xml (deflated 83%) 2025-12-04T12:13:24.4474302Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-093e5ad7fc260f19.xml (deflated 37%) 2025-12-04T12:13:24.4475610Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-60dcb39f38cfdee4.xml (deflated 35%) 2025-12-04T12:13:24.4476925Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-c9c5102c76f0a9a2.xml (deflated 34%) 2025-12-04T12:13:24.4478234Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-312bbf42e62ed9b6.xml (deflated 34%) 2025-12-04T12:13:24.4479707Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-75c9bd860618416d.xml (deflated 36%) 2025-12-04T12:13:24.4480958Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-104493b7b71c2fe7.xml (deflated 36%) 2025-12-04T12:13:24.4482300Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-d6965728c111189f.xml (deflated 35%) 2025-12-04T12:13:24.4483528Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-550261c769de2900.xml (deflated 35%) 2025-12-04T12:13:24.4484755Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-c61f939b9c1212d0.xml (deflated 34%) 2025-12-04T12:13:24.4485976Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-f26443fa9fd7d2be.xml (deflated 34%) 2025-12-04T12:13:24.4487207Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-33f9aeac8ff2d88d.xml (deflated 34%) 2025-12-04T12:13:24.4488432Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-6f219815bc6782ef.xml (deflated 34%) 2025-12-04T12:13:24.4489653Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-1f65ab45ec69dc91.xml (deflated 35%) 2025-12-04T12:13:24.4490952Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a7ef02fe48ea3a40.xml (deflated 34%) 2025-12-04T12:13:24.4492174Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a8622e73769262d0.xml (deflated 34%) 2025-12-04T12:13:24.4493393Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-3ba3606e730f6b82.xml (deflated 34%) 2025-12-04T12:13:24.4494664Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-48cf7919df9353bf.xml (deflated 34%) 2025-12-04T12:13:24.4495882Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-5927afc313986450.xml (deflated 34%) 2025-12-04T12:13:24.4497096Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-e564db12bfa2b80c.xml (deflated 34%) 2025-12-04T12:13:24.4498350Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-17d2371d605f4c7c.xml (deflated 35%) 2025-12-04T12:13:24.4499580Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b0038959c806a90a.xml (deflated 35%) 2025-12-04T12:13:24.4500797Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-5cc7624c76b4d82c.xml (deflated 36%) 2025-12-04T12:13:24.4502017Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-ef399536ef6f172f.xml (deflated 36%) 2025-12-04T12:13:24.4503237Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b77e93b7526c6e31.xml (deflated 36%) 2025-12-04T12:13:24.4504475Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-52e140e51a31b98a.xml (deflated 35%) 2025-12-04T12:13:24.4505708Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b7bc54ed87c9274a.xml (deflated 34%) 2025-12-04T12:13:24.4506931Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-927eadde7027cdb3.xml (deflated 33%) 2025-12-04T12:13:24.4508580Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_mixed_precision/distributed.fsdp.test_fsdp_mixed_precision-c50711adaece2e48.xml (deflated 93%) 2025-12-04T12:13:24.4510024Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-81f85fc9fa3db30b.xml (deflated 37%) 2025-12-04T12:13:24.4511292Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0df1dc2a08194e48.xml (deflated 33%) 2025-12-04T12:13:24.4512552Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-43498c0a89946df9.xml (deflated 34%) 2025-12-04T12:13:24.4513815Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-af4c89e072116764.xml (deflated 34%) 2025-12-04T12:13:24.4515085Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-98f78a29e90359ed.xml (deflated 35%) 2025-12-04T12:13:24.4516351Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-30adfd6c7b8ecf71.xml (deflated 34%) 2025-12-04T12:13:24.4517606Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-581dd59091205273.xml (deflated 34%) 2025-12-04T12:13:24.4518854Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d528c59f12482b1.xml (deflated 35%) 2025-12-04T12:13:24.4520200Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d6315101734e178a.xml (deflated 57%) 2025-12-04T12:13:24.4521373Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d89c8d6fc7b73a1b.xml (deflated 57%) 2025-12-04T12:13:24.4522505Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c23cf3f46128ac0f.xml (deflated 34%) 2025-12-04T12:13:24.4523760Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3a09c39260c6bdcb.xml (deflated 34%) 2025-12-04T12:13:24.4525172Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f486758b2548beb0.xml (deflated 34%) 2025-12-04T12:13:24.4526496Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-36d76263f3b24111.xml (deflated 34%) 2025-12-04T12:13:24.4527758Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f1c94044951be399.xml (deflated 34%) 2025-12-04T12:13:24.4529012Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b9df70724e6dcd35.xml (deflated 34%) 2025-12-04T12:13:24.4530341Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fbd78536cd21cf97.xml (deflated 36%) 2025-12-04T12:13:24.4531596Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ca3fc647503e2182.xml (deflated 34%) 2025-12-04T12:13:24.4532852Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-15e1be913d8837e9.xml (deflated 34%) 2025-12-04T12:13:24.4534102Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8a7b7a30260424a5.xml (deflated 34%) 2025-12-04T12:13:24.4535364Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-13f85e6fb9609878.xml (deflated 36%) 2025-12-04T12:13:24.4536715Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d09ed72d675bfdc8.xml (deflated 36%) 2025-12-04T12:13:24.4537846Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3eb78e74581a557b.xml (deflated 36%) 2025-12-04T12:13:24.4538948Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-90acb82834a56757.xml (deflated 35%) 2025-12-04T12:13:24.4540066Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a1738cfdeff44f2e.xml (deflated 36%) 2025-12-04T12:13:24.4541191Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7936e4b0274d7ad6.xml (deflated 36%) 2025-12-04T12:13:24.4542321Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bd821bd95ef10bd6.xml (deflated 36%) 2025-12-04T12:13:24.4543437Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5839fc6b3294dc3b.xml (deflated 37%) 2025-12-04T12:13:24.4544559Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-be22aa9f1a9dc572.xml (deflated 37%) 2025-12-04T12:13:24.4545686Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-63d98e3d16f90c98.xml (deflated 36%) 2025-12-04T12:13:24.4546800Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-93df4bdbf125060c.xml (deflated 35%) 2025-12-04T12:13:24.4548171Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c1c3a483347ae16e.xml (deflated 35%) 2025-12-04T12:13:24.4549445Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5c6d35f4b8178e66.xml (deflated 37%) 2025-12-04T12:13:24.4550707Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f4a815301947af44.xml (deflated 36%) 2025-12-04T12:13:24.4551982Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ea05d17fa83aaab3.xml (deflated 36%) 2025-12-04T12:13:24.4553334Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c99c7919907a8f51.xml (deflated 35%) 2025-12-04T12:13:24.4554606Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3228075a6adb947.xml (deflated 38%) 2025-12-04T12:13:24.4555874Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5f3dfac751b51f15.xml (deflated 36%) 2025-12-04T12:13:24.4557171Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9836e36a022fec7d.xml (deflated 37%) 2025-12-04T12:13:24.4558430Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-481f5a9e007c92bc.xml (deflated 37%) 2025-12-04T12:13:24.4559805Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8e5333f0f8c7af12.xml (deflated 36%) 2025-12-04T12:13:24.4561103Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-27c65021619c5afb.xml (deflated 35%) 2025-12-04T12:13:24.4562230Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3bc159f1fc89dbaa.xml (deflated 35%) 2025-12-04T12:13:24.4563345Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4183a04f60a93385.xml (deflated 45%) 2025-12-04T12:13:24.4564472Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b6ee0414e522cc1e.xml (deflated 44%) 2025-12-04T12:13:24.4565602Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ba060feb7f6eb8f7.xml (deflated 36%) 2025-12-04T12:13:24.4566725Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c3aa486503236fa6.xml (deflated 37%) 2025-12-04T12:13:24.4567833Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a218cf86ddf03214.xml (deflated 36%) 2025-12-04T12:13:24.4568962Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fb621eae387ef1df.xml (deflated 37%) 2025-12-04T12:13:24.4570094Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-80e6fc2adcb64c0a.xml (deflated 35%) 2025-12-04T12:13:24.4571213Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2323c2353e236ae2.xml (deflated 35%) 2025-12-04T12:13:24.4572324Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-21c4771a3c9fd2ac.xml (deflated 35%) 2025-12-04T12:13:24.4573449Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e704bc72dcc2db74.xml (deflated 36%) 2025-12-04T12:13:24.4574572Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8da80ebf3429571e.xml (deflated 36%) 2025-12-04T12:13:24.4575701Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-aad331620921f562.xml (deflated 36%) 2025-12-04T12:13:24.4576806Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4d876218f2c25b17.xml (deflated 34%) 2025-12-04T12:13:24.4577921Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8a4e259673457968.xml (deflated 34%) 2025-12-04T12:13:24.4579046Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d2017d482dc7eebb.xml (deflated 35%) 2025-12-04T12:13:24.4580169Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-cd8336cfdb34c878.xml (deflated 36%) 2025-12-04T12:13:24.4581275Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-be2011792fadd5cd.xml (deflated 35%) 2025-12-04T12:13:24.4582459Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fd2ba77f6fee2f8a.xml (deflated 35%) 2025-12-04T12:13:24.4583600Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-da6ff71bbbbcfd5b.xml (deflated 34%) 2025-12-04T12:13:24.4584725Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-00c7457a094406e2.xml (deflated 35%) 2025-12-04T12:13:24.4585873Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f38ba7e725b3376c.xml (deflated 34%) 2025-12-04T12:13:24.4586991Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-a85d18d4001a0d1f.xml (deflated 35%) 2025-12-04T12:13:24.4588409Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-77e675aec14c1ef5.xml (deflated 36%) 2025-12-04T12:13:24.4589676Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-679877f4d338af19.xml (deflated 36%) 2025-12-04T12:13:24.4590976Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8d26022f0d5f0e5b.xml (deflated 36%) 2025-12-04T12:13:24.4592232Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6d52baf36617de6c.xml (deflated 34%) 2025-12-04T12:13:24.4593508Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-97cfc974aeae8bab.xml (deflated 34%) 2025-12-04T12:13:24.4594781Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dc857ca5b8949f50.xml (deflated 34%) 2025-12-04T12:13:24.4596040Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-68d634b03e6f1cf2.xml (deflated 35%) 2025-12-04T12:13:24.4597296Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-797d0b27326bd9f0.xml (deflated 35%) 2025-12-04T12:13:24.4598563Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-7f4f2591d711a574.xml (deflated 35%) 2025-12-04T12:13:24.4599934Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-54ee371b40f31a9b.xml (deflated 34%) 2025-12-04T12:13:24.4601194Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bc7d0e9f12eb7a6c.xml (deflated 34%) 2025-12-04T12:13:24.4602313Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4b8aea19c365479d.xml (deflated 34%) 2025-12-04T12:13:24.4603443Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ace037ae618597c7.xml (deflated 34%) 2025-12-04T12:13:24.4604568Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8980b265f0dcd572.xml (deflated 34%) 2025-12-04T12:13:24.4605698Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2f6c447cb3d8d9ba.xml (deflated 34%) 2025-12-04T12:13:24.4606812Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-902c7c5c4fac1c79.xml (deflated 34%) 2025-12-04T12:13:24.4607939Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3399e4ec5bb86d0f.xml (deflated 35%) 2025-12-04T12:13:24.4609068Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ee7e5984f1b0da3e.xml (deflated 42%) 2025-12-04T12:13:24.4610521Z adding: test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-daac7fd3db05a588.xml (deflated 28%) 2025-12-04T12:13:24.4612288Z adding: test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-a07a6d6cccb031bb.xml (deflated 28%) 2025-12-04T12:13:24.4614097Z adding: test/test-reports/dist-mpi-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-7c0282eac181e46a.xml (deflated 27%) 2025-12-04T12:13:24.4615643Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-975ac1cd2ed3ea0d.xml (deflated 27%) 2025-12-04T12:13:24.4616965Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8babbac9d40e37c7.xml (deflated 28%) 2025-12-04T12:13:24.4618318Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4f717ad1d99b753.xml (deflated 27%) 2025-12-04T12:13:24.4619621Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85365fcd3056dbf4.xml (deflated 28%) 2025-12-04T12:13:24.4620937Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-af2626afcb0e4552.xml (deflated 28%) 2025-12-04T12:13:24.4622280Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a30a6d9cd45f8b4.xml (deflated 28%) 2025-12-04T12:13:24.4623730Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6f6e95ed9bc43f3.xml (deflated 28%) 2025-12-04T12:13:24.4625369Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17e902b4247dcc9d.xml (deflated 28%) 2025-12-04T12:13:24.4626830Z adding: test/test-reports/dist-mpi-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3a3b3a8e58fbb071.xml (deflated 28%) 2025-12-04T12:13:24.4628627Z adding: test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-69522635455745bd.xml (deflated 28%) 2025-12-04T12:13:24.4630634Z adding: test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-b1c2edd3b83dd6a9.xml (deflated 28%) 2025-12-04T12:13:24.4632627Z adding: test/test-reports/dist-mpi-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-8fc6dea04b4e72e8.xml (deflated 28%) 2025-12-04T12:13:24.4634357Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-337a55931fe74487.xml (deflated 28%) 2025-12-04T12:13:24.4635843Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-90b3763c34aa2960.xml (deflated 28%) 2025-12-04T12:13:24.4637332Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d0f38b19d35d9f0.xml (deflated 28%) 2025-12-04T12:13:24.4638827Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-62bde0ef8091e7c6.xml (deflated 28%) 2025-12-04T12:13:24.4640319Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-101e5756265aa78e.xml (deflated 28%) 2025-12-04T12:13:24.4641631Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3553a63d7be9bc12.xml (deflated 28%) 2025-12-04T12:13:24.4642950Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4c18a85b55f10e02.xml (deflated 28%) 2025-12-04T12:13:24.4644279Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee7c4f0b0fe742e4.xml (deflated 28%) 2025-12-04T12:13:24.4645599Z adding: test/test-reports/dist-mpi-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a74d4ada0231688.xml (deflated 28%) 2025-12-04T12:13:24.4647230Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-694fabcdc25b908f.xml (deflated 49%) 2025-12-04T12:13:24.4648996Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cc8c47d77c37f668.xml (deflated 49%) 2025-12-04T12:13:24.4650793Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-93164f6c3fb62a8d.xml (deflated 38%) 2025-12-04T12:13:24.4652556Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-55976768b0d94051.xml (deflated 38%) 2025-12-04T12:13:24.4654316Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-ed34111b302d3808.xml (deflated 38%) 2025-12-04T12:13:24.4656096Z adding: test/test-reports/dist-nccl-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-160b76a599e6e82f.xml (deflated 38%) 2025-12-04T12:13:24.4657646Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a57a7c1f6c446d8.xml (deflated 36%) 2025-12-04T12:13:24.4658966Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9a464945bf1177b.xml (deflated 37%) 2025-12-04T12:13:24.4660290Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1497afbb3545da85.xml (deflated 42%) 2025-12-04T12:13:24.4661608Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cbb5e6acdeef21a7.xml (deflated 45%) 2025-12-04T12:13:24.4662928Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e37edf773eeec4cb.xml (deflated 43%) 2025-12-04T12:13:24.4664256Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d9cafccc700b8925.xml (deflated 43%) 2025-12-04T12:13:24.4665579Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4b34ce6e907f1d03.xml (deflated 43%) 2025-12-04T12:13:24.4667116Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1450c3d527670c7.xml (deflated 36%) 2025-12-04T12:13:24.4668805Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-85297e8464ee07d1.xml (deflated 44%) 2025-12-04T12:13:24.4670300Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aff79cf3432fbb8f.xml (deflated 44%) 2025-12-04T12:13:24.4671796Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8141293960cc76b4.xml (deflated 36%) 2025-12-04T12:13:24.4673285Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ed1d3854cb079b5.xml (deflated 36%) 2025-12-04T12:13:24.4674782Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f5ee7b2c181b77f.xml (deflated 36%) 2025-12-04T12:13:24.4676257Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3de0668036a4d46a.xml (deflated 35%) 2025-12-04T12:13:24.4677746Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ed12ff4e601d4a5.xml (deflated 48%) 2025-12-04T12:13:24.4679314Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55742e93980fa69f.xml (deflated 36%) 2025-12-04T12:13:24.4680188Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d1649152d68be9c.xml (deflated 35%) 2025-12-04T12:13:24.4680831Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3c70db6f7d0cc8f.xml (deflated 35%) 2025-12-04T12:13:24.4681600Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b0fb83c60b047974.xml (deflated 47%) 2025-12-04T12:13:24.4682210Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78a8a7c12b3fdbcd.xml (deflated 35%) 2025-12-04T12:13:24.4682806Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88531b1bf42e91d7.xml (deflated 36%) 2025-12-04T12:13:24.4683430Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22e23e45bc80123f.xml (deflated 36%) 2025-12-04T12:13:24.4684042Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f92f6113f8978240.xml (deflated 37%) 2025-12-04T12:13:24.4684642Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15dc7f75169494c0.xml (deflated 35%) 2025-12-04T12:13:24.4685248Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e73ef2b0e35926a2.xml (deflated 35%) 2025-12-04T12:13:24.4685842Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0f598c7195f78ef8.xml (deflated 36%) 2025-12-04T12:13:24.4686440Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-88b5e238fde7cdb9.xml (deflated 35%) 2025-12-04T12:13:24.4687044Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b8880a41f88a6bb0.xml (deflated 45%) 2025-12-04T12:13:24.4687636Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b0298f7abe568d4.xml (deflated 57%) 2025-12-04T12:13:24.4688240Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e4e163fd1daa635.xml (deflated 36%) 2025-12-04T12:13:24.4688837Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9ab104c2f6083ce.xml (deflated 46%) 2025-12-04T12:13:24.4689440Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d76d59daf2e605c7.xml (deflated 46%) 2025-12-04T12:13:24.4690037Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c31c06f41345c5c.xml (deflated 43%) 2025-12-04T12:13:24.4690634Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2a98e316dc116cc1.xml (deflated 35%) 2025-12-04T12:13:24.4691237Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-723af9187a42bd99.xml (deflated 35%) 2025-12-04T12:13:24.4691832Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf92d20898f195ab.xml (deflated 43%) 2025-12-04T12:13:24.4692431Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-524dc1dcdc5a1168.xml (deflated 42%) 2025-12-04T12:13:24.4693024Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98d2197e9fdc8948.xml (deflated 36%) 2025-12-04T12:13:24.4693668Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2102f231775de0a.xml (deflated 44%) 2025-12-04T12:13:24.4694265Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-271cbb7ef1d534a9.xml (deflated 44%) 2025-12-04T12:13:24.4694886Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d42c353b826e766.xml (deflated 36%) 2025-12-04T12:13:24.4695501Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4f9face38efde3d.xml (deflated 36%) 2025-12-04T12:13:24.4696096Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-65eb8dabd26f049f.xml (deflated 35%) 2025-12-04T12:13:24.4696727Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-993c00909ddd3eaa.xml (deflated 35%) 2025-12-04T12:13:24.4697316Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a20300b26842835f.xml (deflated 35%) 2025-12-04T12:13:24.4697905Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8300341a4195c4a2.xml (deflated 36%) 2025-12-04T12:13:24.4698508Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e45287a9f93d9858.xml (deflated 37%) 2025-12-04T12:13:24.4699103Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa4ae74c761f06da.xml (deflated 37%) 2025-12-04T12:13:24.4699707Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b7742dab30c5a9.xml (deflated 35%) 2025-12-04T12:13:24.4700306Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92706bcf8036c15a.xml (deflated 36%) 2025-12-04T12:13:24.4700902Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e2d29ed0b71edf9.xml (deflated 36%) 2025-12-04T12:13:24.4701505Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b82693979bf3f797.xml (deflated 35%) 2025-12-04T12:13:24.4702100Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40f956e1991c429b.xml (deflated 35%) 2025-12-04T12:13:24.4702712Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be8fbd355ae7d4d2.xml (deflated 35%) 2025-12-04T12:13:24.4703310Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00fdf0a71d560046.xml (deflated 36%) 2025-12-04T12:13:24.4703910Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f23f45b0fd1b4e0.xml (deflated 35%) 2025-12-04T12:13:24.4704516Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a704bee28922d105.xml (deflated 44%) 2025-12-04T12:13:24.4705112Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-59aa1d9ad1b952bc.xml (deflated 43%) 2025-12-04T12:13:24.4705718Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f46bcad2881001f4.xml (deflated 35%) 2025-12-04T12:13:24.4706310Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bdf0f48b15d753a.xml (deflated 43%) 2025-12-04T12:13:24.4706983Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3701eb2bf6e59b5c.xml (deflated 43%) 2025-12-04T12:13:24.4707807Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4294fcb622b36e12.xml (deflated 36%) 2025-12-04T12:13:24.4708478Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25f21f2c9cc3f58e.xml (deflated 43%) 2025-12-04T12:13:24.4709199Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a888dfec83025ca2.xml (deflated 45%) 2025-12-04T12:13:24.4709868Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a2971cbec516283.xml (deflated 45%) 2025-12-04T12:13:24.4710545Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-831b908a573477b4.xml (deflated 35%) 2025-12-04T12:13:24.4711251Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-838476cdd85e63de.xml (deflated 36%) 2025-12-04T12:13:24.4711921Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7c1e5f9052a62ed.xml (deflated 44%) 2025-12-04T12:13:24.4712610Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9fe4918eb2ec3fc.xml (deflated 43%) 2025-12-04T12:13:24.4713277Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-18ee4fc76444d2ea.xml (deflated 36%) 2025-12-04T12:13:24.4713959Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf306b33274dddf6.xml (deflated 44%) 2025-12-04T12:13:24.4714634Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c91a26250cab8d8.xml (deflated 36%) 2025-12-04T12:13:24.4715314Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12d1ac9616f3d37b.xml (deflated 36%) 2025-12-04T12:13:24.4715984Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24baa892d95df88d.xml (deflated 44%) 2025-12-04T12:13:24.4716654Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1238a6b3de9abf1.xml (deflated 43%) 2025-12-04T12:13:24.4717330Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5960ed309c5523d7.xml (deflated 35%) 2025-12-04T12:13:24.4717995Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-484220487412d630.xml (deflated 35%) 2025-12-04T12:13:24.4718684Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcdabce46aafa3d7.xml (deflated 36%) 2025-12-04T12:13:24.4719350Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-712b9444633e7909.xml (deflated 36%) 2025-12-04T12:13:24.4720124Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22f63a5524b50ec1.xml (deflated 44%) 2025-12-04T12:13:24.4720732Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9abdeaa3f668748b.xml (deflated 35%) 2025-12-04T12:13:24.4721326Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db023d74613c38df.xml (deflated 43%) 2025-12-04T12:13:24.4721925Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d349e5cd46aba4b9.xml (deflated 44%) 2025-12-04T12:13:24.4722577Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dcab0e823bbcb484.xml (deflated 56%) 2025-12-04T12:13:24.4723174Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-107117b39386c513.xml (deflated 35%) 2025-12-04T12:13:24.4724103Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2a16f55ac9cd547.xml (deflated 42%) 2025-12-04T12:13:24.4724772Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f920c9d47a13120.xml (deflated 43%) 2025-12-04T12:13:24.4725450Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0adedc6075462154.xml (deflated 43%) 2025-12-04T12:13:24.4726188Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51b3a60aa7e4ef26.xml (deflated 45%) 2025-12-04T12:13:24.4726872Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9abab74452896355.xml (deflated 36%) 2025-12-04T12:13:24.4727544Z adding: test/test-reports/dist-nccl-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa158d78c6a1047c.xml (deflated 35%) 2025-12-04T12:13:24.4728491Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-dbafcff6c2749b84.xml (deflated 49%) 2025-12-04T12:13:24.4729435Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-80abfc2e08c5c31a.xml (deflated 49%) 2025-12-04T12:13:24.4730375Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-281d630b2fcf32af.xml (deflated 38%) 2025-12-04T12:13:24.4731315Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-875849d617d7b51e.xml (deflated 38%) 2025-12-04T12:13:24.4732244Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-4cfe0d1200d89b83.xml (deflated 38%) 2025-12-04T12:13:24.4733187Z adding: test/test-reports/dist-nccl-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-366840efd0ab41ce.xml (deflated 38%) 2025-12-04T12:13:24.4733867Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-365036945f26d41b.xml (deflated 36%) 2025-12-04T12:13:24.4734560Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e819ba4cf6e31f8.xml (deflated 37%) 2025-12-04T12:13:24.4735252Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40bc39b1652660d2.xml (deflated 42%) 2025-12-04T12:13:24.4735927Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f9afae11c179070.xml (deflated 45%) 2025-12-04T12:13:24.4736853Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f3c746330754bbc2.xml (deflated 43%) 2025-12-04T12:13:24.4737451Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b68ead304b495067.xml (deflated 43%) 2025-12-04T12:13:24.4738065Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b7886ccd5b9eeb78.xml (deflated 43%) 2025-12-04T12:13:24.4738736Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21598fac9ec33b36.xml (deflated 36%) 2025-12-04T12:13:24.4739337Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45fc07c2582c64de.xml (deflated 44%) 2025-12-04T12:13:24.4739959Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9a64b3d5828bdd7.xml (deflated 44%) 2025-12-04T12:13:24.4740593Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f441409f584e7860.xml (deflated 36%) 2025-12-04T12:13:24.4741208Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-02d51cd92aaffd99.xml (deflated 36%) 2025-12-04T12:13:24.4741840Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd7c20c72a58c490.xml (deflated 36%) 2025-12-04T12:13:24.4742436Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a885f907c0ea15f.xml (deflated 35%) 2025-12-04T12:13:24.4743053Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd6772c504021b1c.xml (deflated 48%) 2025-12-04T12:13:24.4743655Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7b129e835aba47f3.xml (deflated 36%) 2025-12-04T12:13:24.4744266Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dca90ceb118f7f51.xml (deflated 35%) 2025-12-04T12:13:24.4744860Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4c7cdb746d63750.xml (deflated 35%) 2025-12-04T12:13:24.4745467Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b62455b26fd75a3.xml (deflated 47%) 2025-12-04T12:13:24.4746062Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c6c3445d3489ed7.xml (deflated 36%) 2025-12-04T12:13:24.4746667Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc9043c9b1f5531d.xml (deflated 35%) 2025-12-04T12:13:24.4747326Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fe0c34ac3a70b57.xml (deflated 36%) 2025-12-04T12:13:24.4748123Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93b994665f730626.xml (deflated 37%) 2025-12-04T12:13:24.4748808Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e40c62350b42648a.xml (deflated 35%) 2025-12-04T12:13:24.4749483Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3c79494752e7d5ea.xml (deflated 35%) 2025-12-04T12:13:24.4750162Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6369d472a2cdae13.xml (deflated 35%) 2025-12-04T12:13:24.4750839Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-342afb36b7c5d09b.xml (deflated 35%) 2025-12-04T12:13:24.4751515Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f18aaac9bd89cd2c.xml (deflated 46%) 2025-12-04T12:13:24.4752189Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d2615b1994fa3617.xml (deflated 57%) 2025-12-04T12:13:24.4753334Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e62634fc743f9964.xml (deflated 35%) 2025-12-04T12:13:24.4754027Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11b374627b30152b.xml (deflated 46%) 2025-12-04T12:13:24.4754705Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2922c97c2127de2.xml (deflated 46%) 2025-12-04T12:13:24.4755422Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-25e6eb10dce656e7.xml (deflated 43%) 2025-12-04T12:13:24.4756100Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9c2f928358ecde73.xml (deflated 35%) 2025-12-04T12:13:24.4756775Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d84b93ba23264de.xml (deflated 35%) 2025-12-04T12:13:24.4757501Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48132f4775de3019.xml (deflated 43%) 2025-12-04T12:13:24.4758178Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4552be91feabceb2.xml (deflated 43%) 2025-12-04T12:13:24.4758853Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6475011a4c42f67b.xml (deflated 36%) 2025-12-04T12:13:24.4759643Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d893aced2165d7e.xml (deflated 44%) 2025-12-04T12:13:24.4760392Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b44c71018f3811f.xml (deflated 44%) 2025-12-04T12:13:24.4761003Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-26728a48a74f0db0.xml (deflated 36%) 2025-12-04T12:13:24.4761604Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9014cdddbced68f.xml (deflated 36%) 2025-12-04T12:13:24.4762205Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-578fd43f9d90672b.xml (deflated 35%) 2025-12-04T12:13:24.4762805Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-293b2ba538d5c1ee.xml (deflated 35%) 2025-12-04T12:13:24.4763417Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea83e8b183ac592d.xml (deflated 35%) 2025-12-04T12:13:24.4764020Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-436923729a977d14.xml (deflated 36%) 2025-12-04T12:13:24.4764624Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e5f688e692439013.xml (deflated 37%) 2025-12-04T12:13:24.4765227Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4dc08961f281c036.xml (deflated 36%) 2025-12-04T12:13:24.4765824Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da7051c1be9f2e2f.xml (deflated 35%) 2025-12-04T12:13:24.4766432Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a6675bf2f877791e.xml (deflated 36%) 2025-12-04T12:13:24.4767028Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f31832653dbf03f.xml (deflated 35%) 2025-12-04T12:13:24.4767695Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc72cfdb4feaf9b1.xml (deflated 35%) 2025-12-04T12:13:24.4768301Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b4abfc0edb4e297a.xml (deflated 35%) 2025-12-04T12:13:24.4768900Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a979b5e962d58cfe.xml (deflated 35%) 2025-12-04T12:13:24.4769537Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-80387dc09d6e68db.xml (deflated 37%) 2025-12-04T12:13:24.4770136Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec959ea151220e03.xml (deflated 35%) 2025-12-04T12:13:24.4770740Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a47b279193b674dc.xml (deflated 44%) 2025-12-04T12:13:24.4771367Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8aed26ab454245f.xml (deflated 43%) 2025-12-04T12:13:24.4771968Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-498b958aacbfbe1c.xml (deflated 35%) 2025-12-04T12:13:24.4772571Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ca585512d847c09.xml (deflated 43%) 2025-12-04T12:13:24.4773167Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-654cb3e44738170f.xml (deflated 43%) 2025-12-04T12:13:24.4773781Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28d3d63fc6efada0.xml (deflated 35%) 2025-12-04T12:13:24.4774382Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-00ace48f549d318a.xml (deflated 43%) 2025-12-04T12:13:24.4774999Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-673dc73dcfb18eb0.xml (deflated 45%) 2025-12-04T12:13:24.4775601Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfd6f3e2d018dce1.xml (deflated 45%) 2025-12-04T12:13:24.4776202Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa80501c9f66d90c.xml (deflated 36%) 2025-12-04T12:13:24.4776812Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e0dbc7fa11b6547.xml (deflated 36%) 2025-12-04T12:13:24.4777414Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75e1b023ce49c02c.xml (deflated 44%) 2025-12-04T12:13:24.4778069Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09a1c22c519254b6.xml (deflated 43%) 2025-12-04T12:13:24.4778675Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c214051d59b82c2.xml (deflated 36%) 2025-12-04T12:13:24.4779273Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed7ddfe1482b6f0b.xml (deflated 44%) 2025-12-04T12:13:24.4779876Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a95a417b4eefa26.xml (deflated 36%) 2025-12-04T12:13:24.4780489Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a2ef4a32ca2b3dbf.xml (deflated 36%) 2025-12-04T12:13:24.4781095Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c5c9327604cb8f14.xml (deflated 44%) 2025-12-04T12:13:24.4781755Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c38e0bbe495b143b.xml (deflated 43%) 2025-12-04T12:13:24.4782362Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cb5b2f78eac5bfd.xml (deflated 35%) 2025-12-04T12:13:24.4782969Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f44a2b1c1e232df9.xml (deflated 36%) 2025-12-04T12:13:24.4783608Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7f8edc520bdf516e.xml (deflated 36%) 2025-12-04T12:13:24.4784206Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-402e1eebb6506e23.xml (deflated 36%) 2025-12-04T12:13:24.4784820Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d8076bdf0a69595.xml (deflated 44%) 2025-12-04T12:13:24.4785450Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b468bab3124b13b3.xml (deflated 35%) 2025-12-04T12:13:24.4786063Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e50d3219cb7bfbb6.xml (deflated 44%) 2025-12-04T12:13:24.4786660Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53454fb25c79a584.xml (deflated 44%) 2025-12-04T12:13:24.4787323Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6ee8c69fe7ba0518.xml (deflated 56%) 2025-12-04T12:13:24.4788138Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1220c9553e1d6abf.xml (deflated 35%) 2025-12-04T12:13:24.4788820Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f13fc2d140babf7f.xml (deflated 43%) 2025-12-04T12:13:24.4789497Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-557712f590042c69.xml (deflated 43%) 2025-12-04T12:13:24.4790176Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce76f7d0c0c1d9a9.xml (deflated 43%) 2025-12-04T12:13:24.4790859Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-323c51dd40cd5ff1.xml (deflated 45%) 2025-12-04T12:13:24.4791534Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fbb3e53e76b1f52.xml (deflated 36%) 2025-12-04T12:13:24.4792209Z adding: test/test-reports/dist-nccl-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17f25234f4eef787.xml (deflated 35%) 2025-12-04T12:13:24.4793151Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-923db93bebc1ae0d.xml (deflated 37%) 2025-12-04T12:13:24.4794080Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cab701c96fd8d8eb.xml (deflated 37%) 2025-12-04T12:13:24.4795017Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-e799b1e80410ac4c.xml (deflated 49%) 2025-12-04T12:13:24.4795945Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-17c54a2ebe2f0c44.xml (deflated 50%) 2025-12-04T12:13:24.4796930Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-275ce76726773c8f.xml (deflated 51%) 2025-12-04T12:13:24.4797848Z adding: test/test-reports/dist-gloo-init-env/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-086dc66f538a7679.xml (deflated 51%) 2025-12-04T12:13:24.4798522Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51c0d6585102aef7.xml (deflated 36%) 2025-12-04T12:13:24.4799233Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8e2c29cc0f67c6e0.xml (deflated 37%) 2025-12-04T12:13:24.4800028Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e04c95a8c2fac8bc.xml (deflated 42%) 2025-12-04T12:13:24.4800629Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2143f4a3910ea5d.xml (deflated 36%) 2025-12-04T12:13:24.4801251Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a623b45ea2222a4a.xml (deflated 36%) 2025-12-04T12:13:24.4801846Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b489fc84ac56c90a.xml (deflated 43%) 2025-12-04T12:13:24.4802444Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c606f93062a316ee.xml (deflated 43%) 2025-12-04T12:13:24.4803035Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13409fa898261f4d.xml (deflated 45%) 2025-12-04T12:13:24.4803634Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-822c6e3f32fc24f4.xml (deflated 44%) 2025-12-04T12:13:24.4804231Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ee05bfd466991d42.xml (deflated 44%) 2025-12-04T12:13:24.4804829Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-997ca84c08e13a4f.xml (deflated 35%) 2025-12-04T12:13:24.4805423Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1650b0a1cf5101b.xml (deflated 36%) 2025-12-04T12:13:24.4806016Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f17cf2fe88278b7.xml (deflated 36%) 2025-12-04T12:13:24.4806626Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d173d160413f56ff.xml (deflated 35%) 2025-12-04T12:13:24.4807217Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c034757c66530030.xml (deflated 48%) 2025-12-04T12:13:24.4807822Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-19e760457ef6ff92.xml (deflated 36%) 2025-12-04T12:13:24.4808417Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78ba10be7d074c23.xml (deflated 35%) 2025-12-04T12:13:24.4809035Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b4bba108f9cb06d.xml (deflated 35%) 2025-12-04T12:13:24.4809629Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-049ae9466a303b5c.xml (deflated 47%) 2025-12-04T12:13:24.4810234Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-44b7dc32fb33d263.xml (deflated 36%) 2025-12-04T12:13:24.4810827Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75e1e407437545e3.xml (deflated 36%) 2025-12-04T12:13:24.4811473Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70326d0de199e967.xml (deflated 36%) 2025-12-04T12:13:24.4812083Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-537e26a62ece27c3.xml (deflated 45%) 2025-12-04T12:13:24.4812716Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-276b8324a6920515.xml (deflated 35%) 2025-12-04T12:13:24.4813323Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2263791b16c51a6.xml (deflated 35%) 2025-12-04T12:13:24.4813921Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebaf98e72782bada.xml (deflated 36%) 2025-12-04T12:13:24.4814540Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51f4be7c600c130e.xml (deflated 35%) 2025-12-04T12:13:24.4815148Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64a85d7811c12838.xml (deflated 35%) 2025-12-04T12:13:24.4815739Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-325d701d1891cfcd.xml (deflated 57%) 2025-12-04T12:13:24.4816354Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bda5b12f5f6c4582.xml (deflated 35%) 2025-12-04T12:13:24.4816947Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71b41420eeb91960.xml (deflated 36%) 2025-12-04T12:13:24.4817551Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d12eb62cb70e222b.xml (deflated 36%) 2025-12-04T12:13:24.4818154Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28e8d5b3cc52ed16.xml (deflated 35%) 2025-12-04T12:13:24.4818748Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3575bf35b7b1c647.xml (deflated 43%) 2025-12-04T12:13:24.4819353Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4522094cb3ebf4f.xml (deflated 44%) 2025-12-04T12:13:24.4819959Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9492f0ab3ccaa218.xml (deflated 35%) 2025-12-04T12:13:24.4820566Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4010fccffcf7594.xml (deflated 35%) 2025-12-04T12:13:24.4821174Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ffb1b63fa92eef97.xml (deflated 36%) 2025-12-04T12:13:24.4821768Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6439db28f2af0685.xml (deflated 44%) 2025-12-04T12:13:24.4822374Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8c2bb33ba5b4e7cc.xml (deflated 44%) 2025-12-04T12:13:24.4822973Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-95dc0d7d8e2b1b6d.xml (deflated 42%) 2025-12-04T12:13:24.4823783Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cde022d82d2adcaa.xml (deflated 42%) 2025-12-04T12:13:24.4824747Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3ea27d76bf44331.xml (deflated 35%) 2025-12-04T12:13:24.4825531Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d08d23ef2c0e694a.xml (deflated 35%) 2025-12-04T12:13:24.4826203Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-862e9783d4f59986.xml (deflated 35%) 2025-12-04T12:13:24.4826874Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4662846f57df901f.xml (deflated 36%) 2025-12-04T12:13:24.4827666Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-201ee2adc01d4dba.xml (deflated 37%) 2025-12-04T12:13:24.4828336Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-232183083f1e6c18.xml (deflated 36%) 2025-12-04T12:13:24.4829018Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-92d5ea877ee6cee8.xml (deflated 36%) 2025-12-04T12:13:24.4829738Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e7940eab8840d89.xml (deflated 36%) 2025-12-04T12:13:24.4830412Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-55d55fb078c2b684.xml (deflated 36%) 2025-12-04T12:13:24.4831095Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a0928e0ccb8a6b10.xml (deflated 35%) 2025-12-04T12:13:24.4831760Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2664d0ec37e344ff.xml (deflated 43%) 2025-12-04T12:13:24.4832443Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32cc8afe70b3e7da.xml (deflated 35%) 2025-12-04T12:13:24.4833121Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d3b40b7260221d.xml (deflated 37%) 2025-12-04T12:13:24.4833803Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ac3d6524701cbf9.xml (deflated 35%) 2025-12-04T12:13:24.4834469Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-15963edc8117391b.xml (deflated 36%) 2025-12-04T12:13:24.4835144Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc7ad1ec18a7bb1f.xml (deflated 37%) 2025-12-04T12:13:24.4835827Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d4b332b6754f948d.xml (deflated 35%) 2025-12-04T12:13:24.4836504Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6f36e4bfeb187dab.xml (deflated 43%) 2025-12-04T12:13:24.4837189Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-046129b2924de1d6.xml (deflated 35%) 2025-12-04T12:13:24.4837864Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1fb6eb7f4419f2c.xml (deflated 44%) 2025-12-04T12:13:24.4838529Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb907762912a3d99.xml (deflated 35%) 2025-12-04T12:13:24.4839332Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39f1cb056fff7a78.xml (deflated 35%) 2025-12-04T12:13:24.4839923Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-285c65bf39f49db0.xml (deflated 35%) 2025-12-04T12:13:24.4840528Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98105ea07b2795a4.xml (deflated 35%) 2025-12-04T12:13:24.4841180Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ecac8ed92cb8e9f2.xml (deflated 36%) 2025-12-04T12:13:24.4841785Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abee6adb87748222.xml (deflated 36%) 2025-12-04T12:13:24.4842402Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0c43579cac56c01.xml (deflated 43%) 2025-12-04T12:13:24.4842994Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-838afcd07f98645e.xml (deflated 43%) 2025-12-04T12:13:24.4843601Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-efef7c0c73e4a3a4.xml (deflated 44%) 2025-12-04T12:13:24.4844218Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-279a5af71a085b78.xml (deflated 44%) 2025-12-04T12:13:24.4844820Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03480c4b462e8e3c.xml (deflated 44%) 2025-12-04T12:13:24.4845408Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-32560068724b69d2.xml (deflated 50%) 2025-12-04T12:13:24.4846010Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e0917bfdcbaebaa.xml (deflated 44%) 2025-12-04T12:13:24.4846617Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-895373b2e146bc0e.xml (deflated 36%) 2025-12-04T12:13:24.4847206Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c7263c0c2b97dd5.xml (deflated 36%) 2025-12-04T12:13:24.4847812Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-607d80da638aa01c.xml (deflated 36%) 2025-12-04T12:13:24.4848404Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-328e7d3fc5b20fc8.xml (deflated 36%) 2025-12-04T12:13:24.4848998Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3fc9d94de1faf069.xml (deflated 44%) 2025-12-04T12:13:24.4849603Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-725b83e279a2fc0a.xml (deflated 35%) 2025-12-04T12:13:24.4850194Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cd5d50730f4030d9.xml (deflated 36%) 2025-12-04T12:13:24.4850806Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d65188606bc4fb38.xml (deflated 35%) 2025-12-04T12:13:24.4851420Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-118492c3de10278a.xml (deflated 56%) 2025-12-04T12:13:24.4852016Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-892e44a3547470c5.xml (deflated 35%) 2025-12-04T12:13:24.4852612Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c889ae266192a3d2.xml (deflated 35%) 2025-12-04T12:13:24.4853201Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91060a4136f305a0.xml (deflated 43%) 2025-12-04T12:13:24.4853806Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abd0041fabca6f79.xml (deflated 44%) 2025-12-04T12:13:24.4854448Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-28c8471f94c139a5.xml (deflated 36%) 2025-12-04T12:13:24.4855052Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-794deb0b1425e8fa.xml (deflated 36%) 2025-12-04T12:13:24.4855646Z adding: test/test-reports/dist-gloo-init-env/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d2d9e77b4c2f3d8.xml (deflated 35%) 2025-12-04T12:13:24.4856681Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-388ce163b11d3c23.xml (deflated 37%) 2025-12-04T12:13:24.4857564Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-e6a8c3241228684f.xml (deflated 37%) 2025-12-04T12:13:24.4858474Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-cd4a02e6e5b27aca.xml (deflated 49%) 2025-12-04T12:13:24.4859353Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-c2167be978f3dc6a.xml (deflated 50%) 2025-12-04T12:13:24.4860221Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-14585f3cbdd695ec.xml (deflated 51%) 2025-12-04T12:13:24.4861105Z adding: test/test-reports/dist-gloo-init-file/distributed.algorithms.quantization.test_quantization/distributed.algorithms.quantization.test_quantization-bfadba8f7e8ab3cd.xml (deflated 51%) 2025-12-04T12:13:24.4861748Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24ebf91103f6fea1.xml (deflated 36%) 2025-12-04T12:13:24.4862397Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-206af608512f699e.xml (deflated 37%) 2025-12-04T12:13:24.4863214Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8f82acc534672cdf.xml (deflated 42%) 2025-12-04T12:13:24.4863869Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bc0e514e7eb9947.xml (deflated 36%) 2025-12-04T12:13:24.4864539Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5301f1689efddffd.xml (deflated 35%) 2025-12-04T12:13:24.4865197Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39f8f74b2c4bb356.xml (deflated 43%) 2025-12-04T12:13:24.4865861Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4deb287bf0f15502.xml (deflated 43%) 2025-12-04T12:13:24.4866522Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-69b712ded204926d.xml (deflated 45%) 2025-12-04T12:13:24.4867178Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac8d4732ea94303d.xml (deflated 44%) 2025-12-04T12:13:24.4868090Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b15b1dbca96c31dd.xml (deflated 43%) 2025-12-04T12:13:24.4868761Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5235361330778453.xml (deflated 36%) 2025-12-04T12:13:24.4869454Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d366d5954251fcda.xml (deflated 36%) 2025-12-04T12:13:24.4870213Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b47d5bdbdea81123.xml (deflated 35%) 2025-12-04T12:13:24.4870907Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1a24f6ee3b330fe2.xml (deflated 35%) 2025-12-04T12:13:24.4871584Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d6ba4b460a331f8f.xml (deflated 48%) 2025-12-04T12:13:24.4872294Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-676fe7068d5b8cce.xml (deflated 35%) 2025-12-04T12:13:24.4872984Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a360a85ac03dda59.xml (deflated 35%) 2025-12-04T12:13:24.4873656Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f1708cca9dc0e798.xml (deflated 36%) 2025-12-04T12:13:24.4874383Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a8ced54e15b6002.xml (deflated 47%) 2025-12-04T12:13:24.4875057Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0c9833d98fcaa164.xml (deflated 36%) 2025-12-04T12:13:24.4875737Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eec63fa6ac957019.xml (deflated 36%) 2025-12-04T12:13:24.4876436Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9032fc63d800720d.xml (deflated 36%) 2025-12-04T12:13:24.4877112Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c9237cfd956a6ea6.xml (deflated 45%) 2025-12-04T12:13:24.4877809Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d3dac1ff57d770a.xml (deflated 35%) 2025-12-04T12:13:24.4878486Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e71aa694c478fc1e.xml (deflated 35%) 2025-12-04T12:13:24.4879168Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3303d981481b9d1f.xml (deflated 36%) 2025-12-04T12:13:24.4880055Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10e39578cdaa15c7.xml (deflated 35%) 2025-12-04T12:13:24.4880692Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d1b45f0468676a59.xml (deflated 36%) 2025-12-04T12:13:24.4881339Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3e83f111f9b87d25.xml (deflated 57%) 2025-12-04T12:13:24.4881977Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10563eb7a07575a6.xml (deflated 36%) 2025-12-04T12:13:24.4882622Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc37255b8c6ba83b.xml (deflated 36%) 2025-12-04T12:13:24.4883257Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13cdaac161151c73.xml (deflated 36%) 2025-12-04T12:13:24.4883896Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6fc399e73bd2cfea.xml (deflated 35%) 2025-12-04T12:13:24.4884548Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1c09d245ecad9319.xml (deflated 43%) 2025-12-04T12:13:24.4885181Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0e3e1dcb90d6929.xml (deflated 44%) 2025-12-04T12:13:24.4885881Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51c71b398d03a4df.xml (deflated 36%) 2025-12-04T12:13:24.4886516Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3006a46890f44c5.xml (deflated 35%) 2025-12-04T12:13:24.4887183Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-267dee8bb7025f13.xml (deflated 36%) 2025-12-04T12:13:24.4887817Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7a7645ce5728017b.xml (deflated 44%) 2025-12-04T12:13:24.4888460Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-deb9bfbb4bd80c4b.xml (deflated 44%) 2025-12-04T12:13:24.4889131Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9e11a11c30a005f3.xml (deflated 42%) 2025-12-04T12:13:24.4889770Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c85cce560acdb42f.xml (deflated 42%) 2025-12-04T12:13:24.4890414Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96d41dfdc3fe6dba.xml (deflated 35%) 2025-12-04T12:13:24.4891054Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c26119bec135fb9d.xml (deflated 35%) 2025-12-04T12:13:24.4891689Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-114850235e770dcb.xml (deflated 35%) 2025-12-04T12:13:24.4892334Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ed8dfc73ef01d6d.xml (deflated 35%) 2025-12-04T12:13:24.4892975Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49b473d112cc4546.xml (deflated 37%) 2025-12-04T12:13:24.4893626Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc13b9aff20c070d.xml (deflated 36%) 2025-12-04T12:13:24.4894266Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-04e707708d96db67.xml (deflated 36%) 2025-12-04T12:13:24.4895030Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4697a0ab5cd20b70.xml (deflated 35%) 2025-12-04T12:13:24.4895633Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a417a16a93526fa4.xml (deflated 35%) 2025-12-04T12:13:24.4896245Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ddfeed99e9cdec6.xml (deflated 35%) 2025-12-04T12:13:24.4896851Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16b81ab28b107f75.xml (deflated 43%) 2025-12-04T12:13:24.4897451Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cff05e49a4594759.xml (deflated 35%) 2025-12-04T12:13:24.4898061Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db1fdc0a1e7f6276.xml (deflated 36%) 2025-12-04T12:13:24.4898665Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cb2bb364c0ae81e.xml (deflated 35%) 2025-12-04T12:13:24.4899262Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-16f5e50845c16298.xml (deflated 36%) 2025-12-04T12:13:24.4899925Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d791b90d6362f3c0.xml (deflated 37%) 2025-12-04T12:13:24.4900522Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96506a809a7d9377.xml (deflated 35%) 2025-12-04T12:13:24.4901126Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-140f4b27a4309298.xml (deflated 43%) 2025-12-04T12:13:24.4901754Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-541ecd361c5cbd66.xml (deflated 36%) 2025-12-04T12:13:24.4902361Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0707caf97ad64431.xml (deflated 44%) 2025-12-04T12:13:24.4902957Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a41a3afe18e29798.xml (deflated 36%) 2025-12-04T12:13:24.4903584Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-384c7454c26fa694.xml (deflated 36%) 2025-12-04T12:13:24.4904191Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39a728e06646a1d3.xml (deflated 35%) 2025-12-04T12:13:24.4904794Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ddaf75327e52f1c1.xml (deflated 36%) 2025-12-04T12:13:24.4905405Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ec25664c586fadf.xml (deflated 36%) 2025-12-04T12:13:24.4905999Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08df3d74d553659c.xml (deflated 36%) 2025-12-04T12:13:24.4906610Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d31b0daadd5824cb.xml (deflated 43%) 2025-12-04T12:13:24.4907223Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-63eea3de02c3a6ef.xml (deflated 43%) 2025-12-04T12:13:24.4908079Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-656d9998e6c222a4.xml (deflated 44%) 2025-12-04T12:13:24.4908775Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6acaa860eb966f60.xml (deflated 44%) 2025-12-04T12:13:24.4909449Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ce80904a0746c1f8.xml (deflated 44%) 2025-12-04T12:13:24.4910132Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13fade6629382363.xml (deflated 50%) 2025-12-04T12:13:24.4910811Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c48a0039a0795ab.xml (deflated 43%) 2025-12-04T12:13:24.4911481Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1960e6db1730a4a3.xml (deflated 35%) 2025-12-04T12:13:24.4912175Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b68267f25edfed92.xml (deflated 36%) 2025-12-04T12:13:24.4912852Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-011a1d9cca18ee04.xml (deflated 36%) 2025-12-04T12:13:24.4913542Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e9ae608f7e37be8.xml (deflated 36%) 2025-12-04T12:13:24.4914307Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-07300589f39fc3ac.xml (deflated 44%) 2025-12-04T12:13:24.4914987Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8cbc9db131dce2e6.xml (deflated 35%) 2025-12-04T12:13:24.4915675Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-458e56e213b7ad87.xml (deflated 35%) 2025-12-04T12:13:24.4916376Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d87679cc1c529e2d.xml (deflated 35%) 2025-12-04T12:13:24.4917058Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4b08dbbb6915206.xml (deflated 56%) 2025-12-04T12:13:24.4917734Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-30aed6037f140bb7.xml (deflated 35%) 2025-12-04T12:13:24.4918449Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-689b3f622d74137a.xml (deflated 35%) 2025-12-04T12:13:24.4919129Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2cb150e7fa6adefe.xml (deflated 43%) 2025-12-04T12:13:24.4919908Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-773bb593cee0da12.xml (deflated 44%) 2025-12-04T12:13:24.4920648Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e3c6b8d3271f370c.xml (deflated 36%) 2025-12-04T12:13:24.4921248Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7116fa9f879b5f5c.xml (deflated 36%) 2025-12-04T12:13:24.4921854Z adding: test/test-reports/dist-gloo-init-file/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ab516b6bd9172a5a.xml (deflated 36%) 2025-12-04T12:13:24.4942918Z ##[group]Run # Remove any previous usage logs if they exist 2025-12-04T12:13:24.4943118Z # Remove any previous usage logs if they exist 2025-12-04T12:13:24.4943224Z rm -f logs-*.zip 2025-12-04T12:13:24.4943415Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-12-04T12:13:24.4943659Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-12-04T12:13:24.4949645Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:24.4949739Z env: 2025-12-04T12:13:24.4949866Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:24.4949969Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:24.4950156Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:24.4950486Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:24.4950808Z FILE_SUFFIX: test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912 2025-12-04T12:13:24.4950913Z ##[endgroup] 2025-12-04T12:13:24.5000418Z adding: usage_log.txt (deflated 58%) 2025-12-04T12:13:24.5086739Z adding: test/test-reports/distributed.test_dynamo_distributed_2.2_9c1eb87275ed5b08_.log (deflated 87%) 2025-12-04T12:13:24.5092059Z adding: test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_04d990d4b348b53e_.log (deflated 95%) 2025-12-04T12:13:24.5097753Z adding: test/test-reports/distributed.fsdp.test_fsdp_input_1.1_b656a1fd46e2d098_.log (deflated 94%) 2025-12-04T12:13:24.5101014Z adding: test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_e348e090749a4a32_.log (deflated 93%) 2025-12-04T12:13:24.5114158Z adding: test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_b53e20b9533041ba_.log (deflated 93%) 2025-12-04T12:13:24.5167563Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_8c711ca16ddf8d8f_.log (deflated 97%) 2025-12-04T12:13:24.5167998Z adding: test/test-reports/distributed.test_c10d_ucc_1.1_511a2327d2886668_.log (stored 0%) 2025-12-04T12:13:24.5169102Z adding: test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_6287b63322acae31_.log (deflated 65%) 2025-12-04T12:13:24.5169505Z adding: test/test-reports/distributed.test_c10d_ops_nccl_1.1_fe8272077a352f98_.log (deflated 81%) 2025-12-04T12:13:24.5175582Z adding: test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_bd37660784d144d8_.log (deflated 91%) 2025-12-04T12:13:24.5177846Z adding: test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_e759e0b14b68a94b_.log (deflated 83%) 2025-12-04T12:13:24.5184129Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_4b073d51b494bfec_.log (deflated 90%) 2025-12-04T12:13:24.5184924Z adding: test/test-reports/distributed.tensor.test_dtensor_export_1.1_1c7ff51f7e217adf_.log (deflated 75%) 2025-12-04T12:13:24.5185530Z adding: test/test-reports/distributed._tools.test_fake_collectives_1.1_1ec2dba68452f8d1_.log (deflated 55%) 2025-12-04T12:13:24.5186254Z adding: test/test-reports/distributed.test_control_collectives_1.1_1b00fbe408668a80_.log (deflated 77%) 2025-12-04T12:13:24.5188451Z adding: test/test-reports/distributed.test_collective_utils_1.1_39bf695810dbd3dc_.log (deflated 89%) 2025-12-04T12:13:24.5191520Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_a57a4bd8eeed9ab0_.log (deflated 93%) 2025-12-04T12:13:24.5192056Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_dc7c5eed900a4a9e_.log (stored 0%) 2025-12-04T12:13:24.5195463Z adding: test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_858ee0662a857cc8_.log (deflated 94%) 2025-12-04T12:13:24.5195990Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_2349b206b6fb541a_.log (stored 0%) 2025-12-04T12:13:24.5201427Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_324153037e83ebec_.log (deflated 95%) 2025-12-04T12:13:24.5202243Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_07b4c07c44ff8426_.log (deflated 83%) 2025-12-04T12:13:24.5203720Z adding: test/test-reports/distributed._shard.test_sharder_1.1_03b9a7c4139e541b_.log (deflated 88%) 2025-12-04T12:13:24.5204507Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_ce2c4a7056f6918d_.log (deflated 83%) 2025-12-04T12:13:24.5206594Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_01149ea100987574_.log (deflated 92%) 2025-12-04T12:13:24.5207977Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_fb0b7b49049cc977_.log (deflated 90%) 2025-12-04T12:13:24.5208895Z adding: test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_5e35faa3f8e40e8a_.log (deflated 76%) 2025-12-04T12:13:24.5210419Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_4c14b89a54f988ad_.log (deflated 90%) 2025-12-04T12:13:24.5212041Z adding: test/test-reports/distributed.test_c10d_pypg_1.1_264bca3f92d10a9e_.log (deflated 87%) 2025-12-04T12:13:24.5213532Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_f309ef666470aa96_.log (deflated 90%) 2025-12-04T12:13:24.5217785Z adding: test/test-reports/distributed.test_pg_wrapper_1.1_ebb877765db6c2f1_.log (deflated 94%) 2025-12-04T12:13:24.5219271Z adding: test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_cca15586252a4feb_.log (deflated 90%) 2025-12-04T12:13:24.5220456Z adding: test/test-reports/distributed.test_c10d_object_collectives_1.1_2145e99c0da21434_.log (deflated 84%) 2025-12-04T12:13:24.5220979Z adding: test/test-reports/distributed.tensor.debug.test_op_coverage_1.1_0b0be77fb03d384c_.log (deflated 54%) 2025-12-04T12:13:24.5226671Z adding: test/test-reports/distributed.test_c10d_common_1.1_41bda74550716a01_.log (deflated 93%) 2025-12-04T12:13:24.5228753Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_1733371dd5ba4a20_.log (deflated 90%) 2025-12-04T12:13:24.5229435Z adding: test/test-reports/distributed._tools.test_mod_tracker_1.1_4c7520fa7d7771cc_.log (deflated 63%) 2025-12-04T12:13:24.5237488Z adding: test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_79ffb8eaa6b21953_.log (deflated 93%) 2025-12-04T12:13:24.5237994Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_b7e7178e36b6771f_.log (deflated 55%) 2025-12-04T12:13:24.5241958Z adding: test/test-reports/distributed.tensor.test_dtensor_compile_1.1_756cc2654f1e6bbb_.log (deflated 89%) 2025-12-04T12:13:24.5260021Z adding: test/test-reports/distributed.test_c10d_nccl_2.3_3f865a26beb29e8c_.log (deflated 93%) 2025-12-04T12:13:24.5263981Z adding: test/test-reports/distributed.test_aten_comm_compute_reordering_2.2_292d0b2b786a1255_.log (deflated 91%) 2025-12-04T12:13:24.5264490Z adding: test/test-reports/distributed._tools.test_sac_estimator_1.1_687d9b4d32abbd61_.log (deflated 59%) 2025-12-04T12:13:24.5265425Z adding: test/test-reports/distributed._tools.test_memory_tracker_1.1_d73934ab53527d1b_.log (deflated 54%) 2025-12-04T12:13:24.5266169Z adding: test/test-reports/distributed.checkpoint._experimental.test_builder_1.1_08d5e7d49c7ad5da_.log (deflated 69%) 2025-12-04T12:13:24.5267226Z adding: test/test-reports/distributed._composable.test_replicate_with_fsdp_1.1_8cf393c82285846c_.log (deflated 78%) 2025-12-04T12:13:24.5268323Z adding: test/test-reports/distributed.tensor.test_xla_integration_1.1_d107779fabf62ae7_.log (deflated 65%) 2025-12-04T12:13:24.5268858Z adding: test/test-reports/distributed.checkpoint._experimental.test_types_1.1_729af79fa7fcfdd8_.log (deflated 64%) 2025-12-04T12:13:24.5269384Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_cd3116aa947c2691_.log (deflated 52%) 2025-12-04T12:13:24.5270189Z adding: test/test-reports/distributed.tensor.experimental.test_register_sharding_1.1_043cb655dab57eee_.log (deflated 75%) 2025-12-04T12:13:24.5270588Z adding: test/test-reports/distributed.test_backends_1.1_75335fea59b22df5_.log (deflated 57%) 2025-12-04T12:13:24.5271514Z adding: test/test-reports/distributed.tensor.test_experimental_ops_1.1_19a129c7d3ded4d3_.log (deflated 72%) 2025-12-04T12:13:24.5272143Z adding: test/test-reports/distributed.checkpoint.test_quantized_hf_storage_1.1_e66d1a41faed9c49_.log (deflated 61%) 2025-12-04T12:13:24.5289873Z adding: test/test-reports/distributed.rpc.cuda.test_tensorpipe_agent_2.2_022525b628906125_.log (deflated 97%) 2025-12-04T12:13:24.5291002Z adding: test/test-reports/distributed._composable.test_composability.test_pp_composability_1.1_626d47024cd05ebf_.log (deflated 90%) 2025-12-04T12:13:24.5293433Z adding: test/test-reports/distributed.checkpoint.test_async_process_executor_1.1_510a90a3345e7d90_.log (deflated 91%) 2025-12-04T12:13:24.5297192Z adding: test/test-reports/distributed.tensor.test_tensor_ops_1.1_cb32e25d0c49245f_.log (deflated 89%) 2025-12-04T12:13:24.5314850Z adding: test/test-reports/distributed.test_device_mesh_1.1_155d8072915be213_.log (deflated 96%) 2025-12-04T12:13:24.5315296Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_72bde93862166bdc_.log (stored 0%) 2025-12-04T12:13:24.5318035Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_49f5842fb075f2b2_.log (deflated 95%) 2025-12-04T12:13:24.5318702Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_ignore_params_1.1_5b98885cf0fe1d50_.log (deflated 63%) 2025-12-04T12:13:24.5319451Z adding: test/test-reports/distributed.checkpoint._experimental.test_staging_1.1_3d837f3dbe22921b_.log (deflated 71%) 2025-12-04T12:13:24.5322787Z adding: test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_62027c77e09bb2e8_.log (deflated 88%) 2025-12-04T12:13:24.5324864Z adding: test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_16c4b7a17248643a_.log (deflated 89%) 2025-12-04T12:13:24.5326154Z adding: test/test-reports/distributed.tensor.test_embedding_ops_1.1_bb5002b50196bdaf_.log (deflated 82%) 2025-12-04T12:13:24.5327105Z adding: test/test-reports/distributed.tensor.experimental.test_local_map_1.1_25f283957a7af0d4_.log (deflated 77%) 2025-12-04T12:13:24.5328277Z adding: test/test-reports/distributed.test_local_tensor_1.1_2cdd9cdded794bb9_.log (deflated 78%) 2025-12-04T12:13:24.5328969Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_a3025a0aeaaae488_.log (deflated 70%) 2025-12-04T12:13:24.5329906Z adding: test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_0e886b22c4d1b3fa_.log (deflated 77%) 2025-12-04T12:13:24.5331180Z adding: test/test-reports/distributed.pipelining.test_stage_1.1_ed32aa9322f8b97b_.log (deflated 82%) 2025-12-04T12:13:24.5331934Z adding: test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_763ea794449f404f_.log (deflated 62%) 2025-12-04T12:13:24.5332914Z adding: test/test-reports/distributed.checkpoint.test_planner_1.1_44287fa663f3fa73_.log (deflated 75%) 2025-12-04T12:13:24.5333713Z adding: test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_365b28a8675e628f_.log (deflated 62%) 2025-12-04T12:13:24.5335290Z adding: test/test-reports/distributed.pipelining.test_schedule_1.1_4fb1c6d0d88f6090_.log (deflated 87%) 2025-12-04T12:13:24.5336987Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_1f62894d62d48806_.log (deflated 89%) 2025-12-04T12:13:24.5337784Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_e1870d365f721364_.log (deflated 58%) 2025-12-04T12:13:24.5338259Z adding: test/test-reports/distributed.test_run_1.1_048c18016597c35d_.log (deflated 59%) 2025-12-04T12:13:24.5341960Z adding: test/test-reports/distributed.tensor.test_math_ops_1.1_3bc3d9be883f9d0b_.log (deflated 89%) 2025-12-04T12:13:24.5342556Z adding: test/test-reports/distributed.fsdp.test_utils_1.1_849efa648c4a4b77_.log (deflated 66%) 2025-12-04T12:13:24.5345221Z adding: test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_5915eea0acf0b33b_.log (deflated 93%) 2025-12-04T12:13:24.5346948Z adding: test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_c34f896008a30836_.log (deflated 91%) 2025-12-04T12:13:24.5351383Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_4d95b4091bc3b359_.log (deflated 90%) 2025-12-04T12:13:24.5353215Z adding: test/test-reports/distributed.test_c10d_spawn_gloo_1.1_0181e7af55ebaf0a_.log (deflated 92%) 2025-12-04T12:13:24.5354718Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_463e91c12def2aed_.log (deflated 84%) 2025-12-04T12:13:24.5358058Z adding: test/test-reports/distributed.checkpoint.test_pg_transport_1.1_26ddb76dc3b5e83f_.log (deflated 93%) 2025-12-04T12:13:24.5364336Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_37715603be2e29d4_.log (deflated 93%) 2025-12-04T12:13:24.5365519Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_9054fdac15ee60e1_.log (deflated 81%) 2025-12-04T12:13:24.5367153Z adding: test/test-reports/distributed.checkpoint.test_utils_1.1_345919bf14d8d525_.log (deflated 89%) 2025-12-04T12:13:24.5367876Z adding: test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_808cfbd3f4bab105_.log (deflated 74%) 2025-12-04T12:13:24.5369965Z adding: test/test-reports/distributed.checkpoint._experimental.test_checkpoint_process_1.1_35f3f306f7f6005f_.log (deflated 87%) 2025-12-04T12:13:24.5372660Z adding: test/test-reports/distributed.tensor.test_view_ops_1.1_75425393e05735f8_.log (deflated 91%) 2025-12-04T12:13:24.5401317Z adding: test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_1eec504a312e52ef_.log (deflated 97%) 2025-12-04T12:13:24.5401884Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_2b1bd9557bd2920d_.log (deflated 12%) 2025-12-04T12:13:24.5402292Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_ee8af3844830e146_.log (deflated 12%) 2025-12-04T12:13:24.5402758Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_f1f2ba629914f596_.log (deflated 82%) 2025-12-04T12:13:24.5403164Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_f4e4b80156692b3f_.log (deflated 82%) 2025-12-04T12:13:24.5408973Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_c85ca92b15a87ea6_.log (deflated 93%) 2025-12-04T12:13:24.5415296Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_7c6c5eed862dee3b_.log (deflated 93%) 2025-12-04T12:13:24.5422248Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_f3ae3ad69e49db65_.log (deflated 93%) 2025-12-04T12:13:24.5430095Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_2ba023b93e4a551d_.log (deflated 93%) 2025-12-04T12:13:24.5430518Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_ad3d12242976253d_.log (deflated 12%) 2025-12-04T12:13:24.5430985Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_b3816ec15751d402_.log (deflated 12%) 2025-12-04T12:13:24.5431412Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_e692ef08a9a81937_.log (deflated 82%) 2025-12-04T12:13:24.5431943Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_466c47db11e78e35_.log (deflated 82%) 2025-12-04T12:13:24.5441189Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_10357f3c9e30db03_.log (deflated 92%) 2025-12-04T12:13:24.5449817Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_2cc71357f7a921e0_.log (deflated 92%) 2025-12-04T12:13:24.5459342Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_403ffc079b9fc114_.log (deflated 94%) 2025-12-04T12:13:24.5469206Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_6ee2fdeafd5e6202_.log (deflated 94%) 2025-12-04T12:13:24.5469619Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_e096ad8d19f564e6_.log (deflated 12%) 2025-12-04T12:13:24.5470049Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_541f6e7caefa0f4f_.log (deflated 12%) 2025-12-04T12:13:24.5470459Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_7e460d66e8af4d5d_.log (deflated 82%) 2025-12-04T12:13:24.5471056Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_015ee03be2a51a26_.log (deflated 82%) 2025-12-04T12:13:24.5476442Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_ebb0a88e8fde4492_.log (deflated 93%) 2025-12-04T12:13:24.5481698Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_4d339c4268e751fc_.log (deflated 93%) 2025-12-04T12:13:24.5488039Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_2be7bbd7c30a5d09_.log (deflated 93%) 2025-12-04T12:13:24.5488599Z adding: test/test-reports/distributed.nn.jit.test_instantiator_1.1_48df896d98c27f75_.log (deflated 63%) 2025-12-04T12:13:24.5490343Z adding: test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_60479538b31460df_.log (deflated 85%) 2025-12-04T12:13:24.5490966Z adding: test/test-reports/distributed.test_nccl_1.1_6ae7ff7c670cceb0_.log (deflated 73%) 2025-12-04T12:13:24.5494376Z adding: test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_91e245634b3e3a2d_.log (deflated 87%) 2025-12-04T12:13:24.5496535Z adding: test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_d514c850fca6a88f_.log (deflated 91%) 2025-12-04T12:13:24.5498140Z adding: test/test-reports/distributed.test_data_parallel_1.1_917049db61ce8799_.log (deflated 87%) 2025-12-04T12:13:24.5525815Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-12-04T12:13:24.5526046Z # Remove any previous debugging artifacts if they exist 2025-12-04T12:13:24.5526176Z rm -f debug-*.zip 2025-12-04T12:13:24.5526296Z if [ -d 'test/debug' ]; then 2025-12-04T12:13:24.5526473Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-12-04T12:13:24.5526571Z fi 2025-12-04T12:13:24.5532299Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:24.5532513Z env: 2025-12-04T12:13:24.5532632Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:24.5532736Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:24.5532928Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:24.5533258Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:24.5533585Z FILE_SUFFIX: test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912 2025-12-04T12:13:24.5533727Z ##[endgroup] 2025-12-04T12:13:24.5615240Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T12:13:24.5615331Z with: 2025-12-04T12:13:24.5615441Z s3-bucket: gha-artifacts 2025-12-04T12:13:24.5615608Z s3-prefix: pytorch/pytorch/19922768520/1/artifact 2025-12-04T12:13:24.5615710Z retention-days: 14 2025-12-04T12:13:24.5615829Z if-no-files-found: warn 2025-12-04T12:13:24.5615933Z path: test-jsons-*.zip 2025-12-04T12:13:24.5616112Z name: artifact 2025-12-04T12:13:24.5616216Z region: us-east-1 2025-12-04T12:13:24.5616298Z env: 2025-12-04T12:13:24.5616409Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:24.5616516Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:24.5616688Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:24.5616998Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:24.5617092Z ##[endgroup] 2025-12-04T12:13:24.9412371Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T12:13:24.9412925Z With the provided path, there will be 1 file uploaded 2025-12-04T12:13:24.9413444Z Uploading to s3 prefix: pytorch/pytorch/19922768520/1/artifact 2025-12-04T12:13:24.9456165Z Starting upload of test-jsons-test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912.zip 2025-12-04T12:13:25.1087355Z Finished upload of test-jsons-test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912.zip 2025-12-04T12:13:25.1285135Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T12:13:25.1285520Z with: 2025-12-04T12:13:25.1285774Z s3-bucket: gha-artifacts 2025-12-04T12:13:25.1286136Z s3-prefix: pytorch/pytorch/19922768520/1/artifact 2025-12-04T12:13:25.1286536Z retention-days: 14 2025-12-04T12:13:25.1286814Z if-no-files-found: error 2025-12-04T12:13:25.1287131Z path: test-reports-*.zip 2025-12-04T12:13:25.1287432Z name: artifact 2025-12-04T12:13:25.1287679Z region: us-east-1 2025-12-04T12:13:25.1287946Z env: 2025-12-04T12:13:25.1288187Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:25.1288475Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:25.1288836Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:25.1289466Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:25.1290026Z ##[endgroup] 2025-12-04T12:13:25.4759446Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T12:13:25.4760138Z With the provided path, there will be 1 file uploaded 2025-12-04T12:13:25.4760676Z Uploading to s3 prefix: pytorch/pytorch/19922768520/1/artifact 2025-12-04T12:13:25.4803893Z Starting upload of test-reports-test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912.zip 2025-12-04T12:13:25.5813189Z Finished upload of test-reports-test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912.zip 2025-12-04T12:13:25.5989583Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T12:13:25.5989980Z with: 2025-12-04T12:13:25.5990243Z s3-bucket: gha-artifacts 2025-12-04T12:13:25.5990620Z s3-prefix: pytorch/pytorch/19922768520/1/artifact 2025-12-04T12:13:25.5991029Z retention-days: 14 2025-12-04T12:13:25.5991315Z if-no-files-found: ignore 2025-12-04T12:13:25.5991628Z path: logs-*.zip 2025-12-04T12:13:25.5991895Z name: artifact 2025-12-04T12:13:25.5992145Z region: us-east-1 2025-12-04T12:13:25.5992406Z env: 2025-12-04T12:13:25.5992655Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:25.5992951Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:25.5993328Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:25.5994346Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:25.5994863Z ##[endgroup] 2025-12-04T12:13:25.9430374Z NOTE: s3-prefix specified, ignoring name parameter 2025-12-04T12:13:25.9430924Z With the provided path, there will be 1 file uploaded 2025-12-04T12:13:25.9431454Z Uploading to s3 prefix: pytorch/pytorch/19922768520/1/artifact 2025-12-04T12:13:25.9472530Z Starting upload of logs-test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912.zip 2025-12-04T12:13:26.0925192Z Finished upload of logs-test-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu_57116084912.zip 2025-12-04T12:13:26.1097964Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-12-04T12:13:26.1098306Z with: 2025-12-04T12:13:26.1098540Z s3-bucket: gha-artifacts 2025-12-04T12:13:26.1098886Z s3-prefix: pytorch/pytorch/19922768520/1/artifact 2025-12-04T12:13:26.1099241Z retention-days: 14 2025-12-04T12:13:26.1099606Z if-no-files-found: ignore 2025-12-04T12:13:26.1099894Z path: debug-*.zip 2025-12-04T12:13:26.1100142Z name: artifact 2025-12-04T12:13:26.1100386Z region: us-east-1 2025-12-04T12:13:26.1100623Z env: 2025-12-04T12:13:26.1100843Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:26.1101110Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:26.1101443Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:26.1102022Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:26.1102529Z ##[endgroup] 2025-12-04T12:13:26.4476205Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-12-04T12:13:26.4658806Z ##[group]Run # shellcheck disable=SC2156 2025-12-04T12:13:26.4659210Z # shellcheck disable=SC2156 2025-12-04T12:13:26.4659861Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-12-04T12:13:26.4666280Z shell: /usr/bin/bash -e {0} 2025-12-04T12:13:26.4666570Z env: 2025-12-04T12:13:26.4666816Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:26.4667119Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:26.4667579Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:26.4668380Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:26.4668966Z ##[endgroup] 2025-12-04T12:13:26.7837963Z ##[group]Run seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a 2025-12-04T12:13:26.7838553Z with: 2025-12-04T12:13:26.7838950Z name: coredumps-distributed-2-3-lf.linux.g4dn.12xlarge.nvidia.gpu 2025-12-04T12:13:26.7839672Z retention-days: 14 2025-12-04T12:13:26.7839951Z if-no-files-found: ignore 2025-12-04T12:13:26.7840238Z path: ./**/core.[1-9]* 2025-12-04T12:13:26.7840528Z s3-bucket: gha-artifacts 2025-12-04T12:13:26.7840820Z region: us-east-1 2025-12-04T12:13:26.7841069Z env: 2025-12-04T12:13:26.7841284Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:26.7841695Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:26.7842027Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:26.7842593Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:26.7843107Z ##[endgroup] 2025-12-04T12:13:34.3598284Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2025-12-04T12:13:34.3856912Z Prepare all required actions 2025-12-04T12:13:34.3857351Z Getting action download info 2025-12-04T12:13:34.5517339Z Download action repository 'actions/setup-python@v6' (SHA:83679a892e2d95755f2dac6acb0bfd1e9ac5d548) 2025-12-04T12:13:34.8666147Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-12-04T12:13:34.8666566Z with: 2025-12-04T12:13:34.8666814Z job_id: 57116084912 2025-12-04T12:13:34.8667571Z job_name: linux-jammy-cuda12.8-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.g4dn.12xlarge.nvidia.gpu, mem_leak_check) 2025-12-04T12:13:34.8668299Z workflow_name: trunk 2025-12-04T12:13:34.8668596Z workflow_run_id: 19922768520 2025-12-04T12:13:34.8668921Z workflow_attempt: 1 2025-12-04T12:13:34.8669198Z env: 2025-12-04T12:13:34.8669427Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:34.8669737Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:34.8670101Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:34.8670785Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:34.8671353Z ##[endgroup] 2025-12-04T12:13:34.8730347Z ##[group]Run actions/setup-python@v6 2025-12-04T12:13:34.8730706Z with: 2025-12-04T12:13:34.8730961Z python-version: 3.10 2025-12-04T12:13:34.8731251Z check-latest: false 2025-12-04T12:13:34.8731661Z token: *** 2025-12-04T12:13:34.8731934Z update-environment: true 2025-12-04T12:13:34.8732261Z allow-prereleases: false 2025-12-04T12:13:34.8732561Z freethreaded: false 2025-12-04T12:13:34.8732838Z env: 2025-12-04T12:13:34.8733080Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:34.8733469Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:34.8733837Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:34.8734491Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:34.8735059Z ##[endgroup] 2025-12-04T12:13:35.0348163Z ##[group]Installed versions 2025-12-04T12:13:35.0357560Z Version 3.10 was not found in the local cache 2025-12-04T12:13:35.0548009Z (node:444844) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead. 2025-12-04T12:13:35.0548971Z (Use `node --trace-deprecation ...` to show where the warning was created) 2025-12-04T12:13:35.4128294Z ##[error]The version '3.10' with architecture 'x64' was not found for this operating system. The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json 2025-12-04T12:13:35.4287514Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-12-04T12:13:35.4287999Z with: 2025-12-04T12:13:35.4288209Z env: 2025-12-04T12:13:35.4288441Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:35.4288733Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:35.4289067Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:35.4289682Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:35.4290227Z ##[endgroup] 2025-12-04T12:13:35.4306422Z ##[group]Run set -eou pipefail 2025-12-04T12:13:35.4306780Z set -eou pipefail 2025-12-04T12:13:35.4307058Z  2025-12-04T12:13:35.4307567Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-12-04T12:13:35.4308276Z for _ in $(seq 1440); do 2025-12-04T12:13:35.4308663Z  # Break if no ssh session exists anymore 2025-12-04T12:13:35.4309056Z  if [ "$(who)" = "" ]; then 2025-12-04T12:13:35.4309448Z  break 2025-12-04T12:13:35.4309708Z  fi 2025-12-04T12:13:35.4309979Z  echo "." 2025-12-04T12:13:35.4310259Z  sleep 5 2025-12-04T12:13:35.4310519Z done 2025-12-04T12:13:35.4316762Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:35.4317201Z env: 2025-12-04T12:13:35.4317449Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:35.4317748Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:35.4318115Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:35.4318762Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:35.4319336Z ##[endgroup] 2025-12-04T12:13:35.4346182Z Holding runner for 2 hours until all ssh sessions have logged out 2025-12-04T12:13:35.4435702Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T12:13:35.4436340Z # ignore expansion of "docker ps -q" since it could be empty 2025-12-04T12:13:35.4436851Z # shellcheck disable=SC2046 2025-12-04T12:13:35.4437246Z docker stop $(docker ps -q) || true 2025-12-04T12:13:35.4437649Z # Prune all of the docker images 2025-12-04T12:13:35.4438017Z docker system prune -af 2025-12-04T12:13:35.4443724Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:35.4444117Z env: 2025-12-04T12:13:35.4444333Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:35.4444614Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:35.4444939Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:35.4445600Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:35.4446119Z ##[endgroup] 2025-12-04T12:13:46.3518282Z 13d84695993c 2025-12-04T12:13:46.9897883Z Deleted Containers: 2025-12-04T12:13:46.9898396Z 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:46.9898797Z 2025-12-04T12:13:54.4083425Z Deleted Images: 2025-12-04T12:13:54.4084466Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11-f0cd68561080d537ef3d3d6f81b25a6416ad600a 2025-12-04T12:13:54.4086309Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:ba21003510dba4bdeed83df81a56fa468e0ee1b612a9445ae1f402a280804f97 2025-12-04T12:13:54.4087357Z deleted: sha256:add7313791033822205cdb3cf32096534b2cfaa4855bd48119b59000bfe00301 2025-12-04T12:13:54.4088085Z deleted: sha256:85a76b7bf29ad34eb76cce6f46af5d49a58b6272f80f983d5c769e82c7749301 2025-12-04T12:13:54.4088824Z deleted: sha256:0882f3ce59ff5ae30195ee4b059fc713e13eda107a3a7814a4616ac9058a30a4 2025-12-04T12:13:54.4089553Z deleted: sha256:64ba5b9344c11a3e4729136076830b90ac4cf1554046edb1bd4f0784b66ebd9b 2025-12-04T12:13:54.4090268Z deleted: sha256:88213c59cf461a65ab9b6cb07b4195dc9d41b5241c152daa002c7b3112e09124 2025-12-04T12:13:54.4090987Z deleted: sha256:4c0f83afa802ffbc05ebaf1aa50e48a2447c7c295549a6dded80ac63437906ca 2025-12-04T12:13:54.4091898Z deleted: sha256:6f7ec74460e8fb070c8209949095ea3be5f4e2fd69c9f750cd39ac4093f5e64b 2025-12-04T12:13:54.4092635Z deleted: sha256:d6928b0d1021b31942fdcb64e5eb4a34682de66e959dd424ed6ed02c29cd706d 2025-12-04T12:13:54.4093339Z deleted: sha256:4e9fbcb1705a6351bb34dd320558752614308636b94fd9ae6f26063e3deadc0a 2025-12-04T12:13:54.4094145Z deleted: sha256:43aabd0201f48712f21758071352dea029b4de37be08b2e2197706856a9ecbf2 2025-12-04T12:13:54.4094949Z deleted: sha256:940a98dec78303f0548beb1033242a45e9097607ef3e55c8b949b69b73d1b95e 2025-12-04T12:13:54.4095668Z deleted: sha256:d2849fa0e0411cf66e4408831d70e38838afb55b11a80c1c4d8aa0ae7dc9ca40 2025-12-04T12:13:54.4096391Z deleted: sha256:14f40d23c20c7e562623f89deb376520296758bc39dd3c77284049b84ebd8a31 2025-12-04T12:13:54.4097110Z deleted: sha256:a8ccba61f90ca097cb391d0f4fbed0d9f821d06b00e28f7332e9e2dcfcbac4ca 2025-12-04T12:13:54.4097937Z deleted: sha256:91b2060d290547d3b517d4a11d994bbe23f4560b5546cb91918ca1828dde6be1 2025-12-04T12:13:54.4098634Z deleted: sha256:b42a184755715dcfead7fad655a127433541d316d9628f5f730ff17ad5f8071c 2025-12-04T12:13:54.4099345Z deleted: sha256:aa5b4f3c9169061dc3c6da0e677e8a86f11ecb0a3f9fb4861ab3d8c04379775c 2025-12-04T12:13:54.4100041Z deleted: sha256:b4dcf450081a48d77fea0a21b8d810a69c03608a595e754fe7d365058d0579b7 2025-12-04T12:13:54.4100742Z deleted: sha256:4f7fe12d3d4f5bf890c7ada4ce16f17a105472aa6509a778f917dcce2f28174b 2025-12-04T12:13:54.4101447Z deleted: sha256:2d1d5a74182594f9a8553df00fdcfc809dba407bcd6700d667f862cbe9d555ce 2025-12-04T12:13:54.4102147Z deleted: sha256:d901e2f5d449aeed16b727bdcc11fc0e0f6c30c8fc5c39ac7eeac8a74d9d176c 2025-12-04T12:13:54.4102858Z deleted: sha256:a04df2603bd12372c6632469a9a81ebc4a8d677452c250672b9692884fa6a452 2025-12-04T12:13:54.4103550Z deleted: sha256:f438a6b52273a552dc3820a55c74c53a62a0eae9f2a7d21b37125add7d71639f 2025-12-04T12:13:54.4104453Z deleted: sha256:d4b09517e9518d709ac98b0ae6f8446ec9ac51688253607b1fca67aa2c87b3f4 2025-12-04T12:13:54.4105168Z deleted: sha256:c1fa38335237f5e7263e39d3d3de98215bcfbbb12b826955c02e149bf68efd13 2025-12-04T12:13:54.4105889Z deleted: sha256:c898d20a30de901fca74d7611663b17ab48e1726a11e031e40548ed16ee81877 2025-12-04T12:13:54.4106616Z deleted: sha256:3baceec7096518fcc10696feba551639d698b3145c2fc09cac927bb60c0fd751 2025-12-04T12:13:54.4107427Z deleted: sha256:5245aaaa3d5c3a19f76b9a6c920bd82d1a0ff5289f87c8c109652089709d9b3b 2025-12-04T12:13:54.4108337Z deleted: sha256:f05cc789b95246938c377f474c41187965b89ceac0250e7d5124bec32153f447 2025-12-04T12:13:54.4109088Z deleted: sha256:07ec4fc008de4e7a2c794ec7094cc72e0d287c04c8b2156163aee0bae147fe2d 2025-12-04T12:13:54.4109953Z deleted: sha256:c6302601ad5fde573c1f8c900250478fca7fdc6907d8fd4fae651b94b4d9264d 2025-12-04T12:13:54.4110697Z deleted: sha256:cc5e955ee1dc54931f02606c5ea87aae14f03b5d764492be611480ab041f2882 2025-12-04T12:13:54.4111446Z deleted: sha256:f21c03518996d98452338f4e80bcfd9b139a1dab155f4830be0d3f623035269f 2025-12-04T12:13:54.4112202Z deleted: sha256:519ca6f1279f7886f25f0005527cfa627deebbc5b7d7cdbfa7ef962bcfc4c26d 2025-12-04T12:13:54.4112947Z deleted: sha256:0ef990495216807d0175b192045be3f617e72331bc373b3434807f41bf69168d 2025-12-04T12:13:54.4113709Z deleted: sha256:7093edf7319e1f0e01654c3224e32c8dede5b948d106e0b9b03cbf0bb1091e33 2025-12-04T12:13:54.4114445Z deleted: sha256:c478161e058e2f4041555c3e880b95ee1ee047938dc58549a3a88135740996ae 2025-12-04T12:13:54.4115190Z deleted: sha256:9bb853b0d938cd7c36a80ce8ee40653f2c0ff92719209b11beb03acc8855ce3e 2025-12-04T12:13:54.4115928Z deleted: sha256:fdf2ace71a78ce6910ef9c4b073c195531da47022443b606bb92dcd6499b6afc 2025-12-04T12:13:54.4116681Z deleted: sha256:576c2b3770d871937d3cfb7014328bcb4bd1aed0c28bc438764b3bfdac4c1ac2 2025-12-04T12:13:54.4117440Z deleted: sha256:878e92b9cb82de09ac14a9d5f3f7bc2411a799b6f54d0d64b78c2bb4d1fdc0fc 2025-12-04T12:13:54.4118189Z deleted: sha256:85c8c3b98b65a6695f988a10cc66c981d73a3ef03eda15b8e14d227b50b56300 2025-12-04T12:13:54.4119045Z deleted: sha256:ce2ab3ba07794f9ee95d6ea7de6dcd3d2aed96561f9a79192dd56ca5bf29313a 2025-12-04T12:13:54.4119876Z deleted: sha256:37a6e12976ca957286977e696e63012ab9821214b0483fe1a48d29dcb280508a 2025-12-04T12:13:54.4120603Z deleted: sha256:cd1d5d3dd7038144ca6fe961c0d4c8e705625ae0c36190ba8b3e9602abedad19 2025-12-04T12:13:54.4121318Z deleted: sha256:0e707276e0be2e0008b86d594fadc0d16444d66c4fb7227c56f144cbb3c2affd 2025-12-04T12:13:54.4122046Z deleted: sha256:22d4aad6a2ada91b341c1225a0f314042b8aeabef7568c5c019709b058bf070b 2025-12-04T12:13:54.4122841Z deleted: sha256:ee4adacf4e0933131d0275eddad406b3c8147e6cf07a292b99f1aff4b5355f33 2025-12-04T12:13:54.4123807Z deleted: sha256:43da0b9e7c0e18403dcb834e53628dc7c970ccb2dbd091878c0d7c0170dbc97f 2025-12-04T12:13:54.4124715Z deleted: sha256:00571684bdcd75beda15eb7d4e79b5458bc914350f9bb4d87fcdc97ad15e0da1 2025-12-04T12:13:54.4125464Z deleted: sha256:41615f09950259f1d75e82ef35b6fc53b18fe71ebff143744cfd51009d04349e 2025-12-04T12:13:54.4126209Z deleted: sha256:75ab34d2eed3c7915467a506ab6dab2711918fbabe94add2fb5c62780221ab0c 2025-12-04T12:13:54.4126971Z deleted: sha256:0a39ef2bebf44c1c3893d1e5fb42dad48b8fac7ca673141267ee967f85455e89 2025-12-04T12:13:54.4127707Z deleted: sha256:9b7d024e48ba1f9824a54597621b1b062cbc4aa41a77d81ca538d6b5c24a612c 2025-12-04T12:13:54.4128440Z deleted: sha256:392257172de6434c271bd93394218a91e9aa86d7c18abc2f2759317b9d5fb6de 2025-12-04T12:13:54.4129161Z deleted: sha256:6c3232860b930866a463a356124fc392c7e5f04895695229257e8c3e8a02711d 2025-12-04T12:13:54.4129881Z deleted: sha256:63dd55b807215e2fa6c715419ac0c5072d02dddc848dbf74bb7e77b906b5eaed 2025-12-04T12:13:54.4130625Z deleted: sha256:07a8738c1b4584db72ed9aa60f5274321eb0ba16263450da3a75df8326ebc25f 2025-12-04T12:13:54.4131357Z deleted: sha256:053fe2965b01281d12040ec1893e0d1aa77362a49ea9a1067402272c69dad9f5 2025-12-04T12:13:54.4132098Z deleted: sha256:7857fb5eb181c4e80262ecab60bdd3c266cf3d1409ceb76c05882609b416a8d3 2025-12-04T12:13:54.4132832Z deleted: sha256:752528477fc99089de3bd2c6da7b30cf34f2e901fe06d8fcfe685b411461e883 2025-12-04T12:13:54.4133578Z deleted: sha256:cce0210e2f4b042601813df03aa294a86b0c668fcfc75f4c63f6fa12b2952e15 2025-12-04T12:13:54.4134324Z deleted: sha256:f2bb405a26705ecd12d21380d26d9355d01db3a2175080fbdb468f2b5a25a76c 2025-12-04T12:13:54.4135073Z deleted: sha256:ad430120d4ffbaf97cd8d6de6ea8eefa4a8f80ec45f0b176c6b26bff0970fd33 2025-12-04T12:13:54.4135966Z deleted: sha256:225a4910baea7cc540ed43eeac75046293800ab0b8e0192b51e991c8cb50bcf3 2025-12-04T12:13:54.4136794Z deleted: sha256:a259945b0c3507f049fbac10fb3d3ffe43d45e83c91b80ae8cd1dafb855ad83c 2025-12-04T12:13:54.4137497Z deleted: sha256:862a98881b1d5adad5c21d01602773b894794097de80964ef8f47bcaadb43255 2025-12-04T12:13:54.4138175Z deleted: sha256:1cf6d3c8b6c2694b79a2d08719594903811c330a36a4c7a8a7153a350b53d292 2025-12-04T12:13:54.4138951Z deleted: sha256:232a1ae8b0fee817ff7838bb5986a2f38377d3b1dbbf5217b576af0f953b0844 2025-12-04T12:13:54.4139658Z deleted: sha256:c72c5705dabd6314423dd7d4fb260a20d5d9886b2ebce60d19e9d78c4a2335c2 2025-12-04T12:13:54.4140343Z deleted: sha256:296734cf81fd92c913884d058908598424ffe072676e38de289bbab83768c7bd 2025-12-04T12:13:54.4141037Z deleted: sha256:7c76040481b889847a1804021aeff07547eaa4ee706d6137db218d497a8fd9c1 2025-12-04T12:13:54.4141789Z deleted: sha256:d5e293f5b354e8cbcc6de893ea72cc632b02d8fdfbb08ec3127c4e9662f3ebff 2025-12-04T12:13:54.4142499Z deleted: sha256:f35a64e429c88e249645090f21fbe7dae108d98e0ab4ea13184f24b3fd66c315 2025-12-04T12:13:54.4143192Z deleted: sha256:ce6ae8d595c8e69115c51b1ce4f9a9158795d7b863b1cb53f21c39a87974d41b 2025-12-04T12:13:54.4143903Z deleted: sha256:8941abaee59400fb9b3a60765fea4a1fc2a6a447467a6d983e84c7f72494a450 2025-12-04T12:13:54.4144618Z deleted: sha256:ef53c29a9a2c2bc80ffdb9bfaf92842436b5755ec1ce828b9d11e5e27d656ea1 2025-12-04T12:13:54.4145347Z deleted: sha256:7a347fb0acb43f1c814f8c8ff21185e8b5cf64d7bc5988cea060f77d906e08b5 2025-12-04T12:13:54.4146040Z deleted: sha256:cc855dc9be79496e15175569dced2d13477e50b077a5fd3945f9bf50018880c1 2025-12-04T12:13:54.4146748Z deleted: sha256:f7a9946ada3d4786658bc0b643808bb32a9a45e4e90e30dc43ee19e2dbe24024 2025-12-04T12:13:54.4147780Z deleted: sha256:c22a9215f62812c1d2e32827f5221ff556c5b6702aadbdab6b87b8293f19635e 2025-12-04T12:13:54.4148519Z deleted: sha256:959a56746620012e37c1def1a83c5afb1e7c0adc59b021a28beb53c24df98032 2025-12-04T12:13:54.4149277Z deleted: sha256:31a0fff0695bf6100c17954be72eab2095b466d559c75c3faf2a17d8c41e6ebe 2025-12-04T12:13:54.4150020Z deleted: sha256:c15e2b5241b9e55af1b2593e544391b4b44d0505e6528e8f12425136e93b424c 2025-12-04T12:13:54.4150758Z deleted: sha256:73974f74b436f39a2fdb6461b1e3f7c3e41c73325776fa71d16b942a5b4a365b 2025-12-04T12:13:54.4151368Z untagged: public.ecr.aws/docker/library/python:3.13 2025-12-04T12:13:54.4152198Z untagged: public.ecr.aws/docker/library/python@sha256:3f986299a7b8b44b0d8cf9bda2b22361ce5c3058ef5d7cb17fb7452506680ab0 2025-12-04T12:13:54.4153177Z deleted: sha256:44438aecfedf7b6086fce506dae0db5ba7fc0027f9b743f1a75a6b5cbc7de70a 2025-12-04T12:13:54.4153928Z deleted: sha256:6f09a1f5d8a107c2532fbd116e75116cb75fa77b1a7d72d3bdf1ac12de152acd 2025-12-04T12:13:54.4154689Z deleted: sha256:fe5f3ac0be086125eb1e3cd10cc33e8e426f4e079381f7ce5a987b626e99fa67 2025-12-04T12:13:54.4155443Z deleted: sha256:79dd2061a22cf919cfc4f1f02704bfda09afadb017265e670ee54441d296c06c 2025-12-04T12:13:54.4156201Z deleted: sha256:9447ad402aafdbee17e999b0ec84ad89c2646dbebf054d469d4f8bee77f66212 2025-12-04T12:13:54.4156933Z deleted: sha256:7a4909f3c1975be52292f53107495ee1b41c17494918767ccedf1cf1688ae318 2025-12-04T12:13:54.4157659Z deleted: sha256:3474923d97f1f498237650a7d51bd4aea37d5e6b9d8a778777920584af5dd560 2025-12-04T12:13:54.4158393Z deleted: sha256:683afd1773444401a9cbd24842ee5d9154a11abb4fab63ddea5c03df788597ee 2025-12-04T12:13:54.4158833Z 2025-12-04T12:13:54.4158982Z Total reclaimed space: 35.58GB 2025-12-04T12:13:54.4191547Z ##[group]Run set +e 2025-12-04T12:13:54.4191928Z set +e 2025-12-04T12:13:54.4192195Z set -x 2025-12-04T12:13:54.4192438Z  2025-12-04T12:13:54.4192686Z nvidia-smi 2025-12-04T12:13:54.4193217Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2025-12-04T12:13:54.4194033Z # the case where the driver has already crashed as it still can get the driver version 2025-12-04T12:13:54.4194816Z # and some basic information like the bus ID. However, the rest of the information 2025-12-04T12:13:54.4195422Z # would be missing (ERR!), for example: 2025-12-04T12:13:54.4195798Z # 2025-12-04T12:13:54.4196155Z # +-----------------------------------------------------------------------------+ 2025-12-04T12:13:54.4196767Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2025-12-04T12:13:54.4197510Z # |-------------------------------+----------------------+----------------------+ 2025-12-04T12:13:54.4198139Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T12:13:54.4198813Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2025-12-04T12:13:54.4199379Z # | | | MIG M. | 2025-12-04T12:13:54.4199971Z # |===============================+======================+======================| 2025-12-04T12:13:54.4200427Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2025-12-04T12:13:54.4200935Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2025-12-04T12:13:54.4201387Z # | | | ERR! | 2025-12-04T12:13:54.4201840Z # +-------------------------------+----------------------+----------------------+ 2025-12-04T12:13:54.4202245Z # 2025-12-04T12:13:54.4202565Z # +-----------------------------------------------------------------------------+ 2025-12-04T12:13:54.4203036Z # | Processes: | 2025-12-04T12:13:54.4203544Z # | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T12:13:54.4204022Z # | ID ID Usage | 2025-12-04T12:13:54.4204422Z # |=============================================================================| 2025-12-04T12:13:54.4204856Z # +-----------------------------------------------------------------------------+ 2025-12-04T12:13:54.4205260Z # 2025-12-04T12:13:54.4205665Z # This should be reported as a failure instead as it will guarantee to fail when 2025-12-04T12:13:54.4206202Z # Docker tries to run with --gpus all 2025-12-04T12:13:54.4206526Z # 2025-12-04T12:13:54.4206907Z # So, the correct check here is to query one of the missing piece of info like 2025-12-04T12:13:54.4207463Z # GPU name, so that the command can fail accordingly 2025-12-04T12:13:54.4207965Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-12-04T12:13:54.4208416Z NVIDIA_SMI_STATUS=$? 2025-12-04T12:13:54.4208702Z  2025-12-04T12:13:54.4209162Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2025-12-04T12:13:54.4209837Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2025-12-04T12:13:54.4210460Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2025-12-04T12:13:54.4210990Z  .github/scripts/stop_runner_service.sh 2025-12-04T12:13:54.4211320Z fi 2025-12-04T12:13:54.4211540Z  2025-12-04T12:13:54.4212046Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2025-12-04T12:13:54.4212705Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2025-12-04T12:13:54.4213241Z # https://github.com/pytorch/test-infra/issues/4000 2025-12-04T12:13:54.4213696Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2025-12-04T12:13:54.4214066Z NVIDIA_SMI_STATUS=$? 2025-12-04T12:13:54.4214332Z  2025-12-04T12:13:54.4214789Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2025-12-04T12:13:54.4215458Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2025-12-04T12:13:54.4216071Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2025-12-04T12:13:54.4216591Z  .github/scripts/stop_runner_service.sh 2025-12-04T12:13:54.4216970Z fi 2025-12-04T12:13:54.4217193Z  2025-12-04T12:13:54.4217437Z # Check the GPU count to be a power of 2 2025-12-04T12:13:54.4218020Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2025-12-04T12:13:54.4218802Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2025-12-04T12:13:54.4219397Z  .github/scripts/stop_runner_service.sh 2025-12-04T12:13:54.4219757Z fi 2025-12-04T12:13:54.4229546Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:54.4229993Z env: 2025-12-04T12:13:54.4230242Z GIT_DEFAULT_BRANCH: main 2025-12-04T12:13:54.4230555Z HAS_NVIDIA_GPU: true 2025-12-04T12:13:54.4230923Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-12-04T12:13:54.4231577Z DOCKER_CONTAINER_ID: 13d84695993c9de4138d4c12fc49728deb9b0ff40feb38a201f37f417d642852 2025-12-04T12:13:54.4232155Z ##[endgroup] 2025-12-04T12:13:54.4261432Z + nvidia-smi 2025-12-04T12:13:54.4718153Z Thu Dec 4 12:13:54 2025 2025-12-04T12:13:54.4718673Z +-----------------------------------------------------------------------------------------+ 2025-12-04T12:13:54.4719298Z | NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 | 2025-12-04T12:13:54.4720023Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T12:13:54.4720654Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-12-04T12:13:54.4721306Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-12-04T12:13:54.4721824Z | | | MIG M. | 2025-12-04T12:13:54.4722235Z |=========================================+========================+======================| 2025-12-04T12:13:54.5357197Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2025-12-04T12:13:54.5357775Z | N/A 26C P8 15W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T12:13:54.5358267Z | | | N/A | 2025-12-04T12:13:54.5358773Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T12:13:54.5359327Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2025-12-04T12:13:54.5359974Z | N/A 27C P8 13W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T12:13:54.5360420Z | | | N/A | 2025-12-04T12:13:54.5360902Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T12:13:54.5361428Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2025-12-04T12:13:54.5362178Z | N/A 26C P8 13W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T12:13:54.5362634Z | | | N/A | 2025-12-04T12:13:54.5363115Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T12:13:54.5363638Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2025-12-04T12:13:54.5364134Z | N/A 28C P8 13W / 70W | 0MiB / 15360MiB | 0% Default | 2025-12-04T12:13:54.5364591Z | | | N/A | 2025-12-04T12:13:54.5365072Z +-----------------------------------------+------------------------+----------------------+ 2025-12-04T12:13:54.5365818Z 2025-12-04T12:13:54.5366043Z +-----------------------------------------------------------------------------------------+ 2025-12-04T12:13:54.5366635Z | Processes: | 2025-12-04T12:13:54.5367182Z | GPU GI CI PID Type Process name GPU Memory | 2025-12-04T12:13:54.5367692Z | ID ID Usage | 2025-12-04T12:13:54.5368117Z |=========================================================================================| 2025-12-04T12:13:54.5388432Z | No running processes found | 2025-12-04T12:13:54.5389148Z +-----------------------------------------------------------------------------------------+ 2025-12-04T12:13:55.1915312Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-12-04T12:13:55.2103500Z Tesla T4 2025-12-04T12:13:55.2389534Z + NVIDIA_SMI_STATUS=0 2025-12-04T12:13:55.2389867Z + '[' 0 -ne 0 ']' 2025-12-04T12:13:55.2395081Z ++ nvidia-smi --list-gpus 2025-12-04T12:13:55.2396236Z ++ wc -l 2025-12-04T12:13:55.2872712Z + GPU_COUNT=4 2025-12-04T12:13:55.2873014Z + NVIDIA_SMI_STATUS=0 2025-12-04T12:13:55.2873321Z + '[' 0 -ne 0 ']' 2025-12-04T12:13:55.2873584Z + '[' 4 -le 8 ']' 2025-12-04T12:13:55.2873820Z + '[' 4 -ne 1 ']' 2025-12-04T12:13:55.2874070Z + '[' 4 -ne 2 ']' 2025-12-04T12:13:55.2874323Z + '[' 4 -ne 4 ']' 2025-12-04T12:13:55.2946405Z Post job cleanup. 2025-12-04T12:13:55.3027637Z Post job cleanup. 2025-12-04T12:13:55.3076470Z Post job cleanup. 2025-12-04T12:13:55.4097957Z [command]/usr/bin/git version 2025-12-04T12:13:55.4139485Z git version 2.50.1 2025-12-04T12:13:55.4177125Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/4b9e32f3-e8e1-4342-afc9-cb87f3e4efda/.gitconfig' 2025-12-04T12:13:55.4187104Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4b9e32f3-e8e1-4342-afc9-cb87f3e4efda' before making global git config changes 2025-12-04T12:13:55.4188530Z Adding repository directory to the temporary git global config as a safe directory 2025-12-04T12:13:55.4192722Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-12-04T12:13:55.4239147Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-12-04T12:13:55.4278414Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-12-04T12:13:55.4610807Z Entering 'android/libs/fbjni' 2025-12-04T12:13:55.4671796Z Entering 'third_party/FP16' 2025-12-04T12:13:55.4729683Z Entering 'third_party/FXdiv' 2025-12-04T12:13:55.4789176Z Entering 'third_party/NNPACK' 2025-12-04T12:13:55.4846794Z Entering 'third_party/NVTX' 2025-12-04T12:13:55.4909481Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T12:13:55.4967655Z Entering 'third_party/XNNPACK' 2025-12-04T12:13:55.5046731Z Entering 'third_party/aiter' 2025-12-04T12:13:55.5109124Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T12:13:55.5178279Z Entering 'third_party/benchmark' 2025-12-04T12:13:55.5236634Z Entering 'third_party/composable_kernel' 2025-12-04T12:13:55.5305742Z Entering 'third_party/cpp-httplib' 2025-12-04T12:13:55.5367598Z Entering 'third_party/cpuinfo' 2025-12-04T12:13:55.5431854Z Entering 'third_party/cudnn_frontend' 2025-12-04T12:13:55.5489719Z Entering 'third_party/cutlass' 2025-12-04T12:13:55.5557481Z Entering 'third_party/fbgemm' 2025-12-04T12:13:55.5617632Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T12:13:55.5677400Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T12:13:55.5745093Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T12:13:55.5802560Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T12:13:55.5870838Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T12:13:55.5929033Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T12:13:55.5989080Z Entering 'third_party/fbgemm/external/json' 2025-12-04T12:13:55.6049954Z Entering 'third_party/flash-attention' 2025-12-04T12:13:55.6109365Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T12:13:55.6173484Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T12:13:55.6244786Z Entering 'third_party/flatbuffers' 2025-12-04T12:13:55.6309159Z Entering 'third_party/fmt' 2025-12-04T12:13:55.6370214Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T12:13:55.6428727Z Entering 'third_party/gloo' 2025-12-04T12:13:55.6488533Z Entering 'third_party/googletest' 2025-12-04T12:13:55.6547610Z Entering 'third_party/ideep' 2025-12-04T12:13:55.6606254Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T12:13:55.6670828Z Entering 'third_party/ittapi' 2025-12-04T12:13:55.6729867Z Entering 'third_party/kineto' 2025-12-04T12:13:55.6789354Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T12:13:55.6847770Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T12:13:55.6909178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T12:13:55.6968291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T12:13:55.7026127Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T12:13:55.7087085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T12:13:55.7148563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T12:13:55.7208186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T12:13:55.7267833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T12:13:55.7326263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T12:13:55.7386180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T12:13:55.7444322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T12:13:55.7507021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T12:13:55.7573815Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T12:13:55.7631698Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T12:13:55.7691638Z Entering 'third_party/kleidiai' 2025-12-04T12:13:55.7750585Z Entering 'third_party/mimalloc' 2025-12-04T12:13:55.7810492Z Entering 'third_party/nlohmann' 2025-12-04T12:13:55.7870910Z Entering 'third_party/onnx' 2025-12-04T12:13:55.7949352Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T12:13:55.8011431Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T12:13:55.8071947Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T12:13:55.8130740Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T12:13:55.8191131Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T12:13:55.8247715Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T12:13:55.8306419Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T12:13:55.8365669Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T12:13:55.8425666Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T12:13:55.8484593Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T12:13:55.8548135Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T12:13:55.8609305Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T12:13:55.8688670Z Entering 'third_party/pocketfft' 2025-12-04T12:13:55.8747017Z Entering 'third_party/protobuf' 2025-12-04T12:13:55.8810335Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T12:13:55.8867874Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T12:13:55.8927200Z Entering 'third_party/psimd' 2025-12-04T12:13:55.8989287Z Entering 'third_party/pthreadpool' 2025-12-04T12:13:55.9046146Z Entering 'third_party/pybind11' 2025-12-04T12:13:55.9105508Z Entering 'third_party/python-peachpy' 2025-12-04T12:13:55.9165644Z Entering 'third_party/sleef' 2025-12-04T12:13:55.9224762Z Entering 'third_party/tensorpipe' 2025-12-04T12:13:55.9290751Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T12:13:55.9355411Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T12:13:55.9411698Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T12:13:55.9469129Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T12:13:55.9525347Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T12:13:55.9610597Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-12-04T12:13:55.9634190Z http.https://github.com/.extraheader 2025-12-04T12:13:55.9643923Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-12-04T12:13:55.9674111Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-12-04T12:13:55.9996979Z Entering 'android/libs/fbjni' 2025-12-04T12:13:56.0036288Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0071603Z Entering 'third_party/FP16' 2025-12-04T12:13:56.0109792Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0146973Z Entering 'third_party/FXdiv' 2025-12-04T12:13:56.0187146Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0223390Z Entering 'third_party/NNPACK' 2025-12-04T12:13:56.0262733Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0300603Z Entering 'third_party/NVTX' 2025-12-04T12:13:56.0341401Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0378150Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T12:13:56.0418919Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0455199Z Entering 'third_party/XNNPACK' 2025-12-04T12:13:56.0495655Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0547803Z Entering 'third_party/aiter' 2025-12-04T12:13:56.0587092Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0627323Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T12:13:56.0665495Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0711321Z Entering 'third_party/benchmark' 2025-12-04T12:13:56.0751773Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0790530Z Entering 'third_party/composable_kernel' 2025-12-04T12:13:56.0828664Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0882240Z Entering 'third_party/cpp-httplib' 2025-12-04T12:13:56.0919507Z http.https://github.com/.extraheader 2025-12-04T12:13:56.0958419Z Entering 'third_party/cpuinfo' 2025-12-04T12:13:56.0996485Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1042567Z Entering 'third_party/cudnn_frontend' 2025-12-04T12:13:56.1083402Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1118396Z Entering 'third_party/cutlass' 2025-12-04T12:13:56.1157802Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1209149Z Entering 'third_party/fbgemm' 2025-12-04T12:13:56.1249776Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1289934Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T12:13:56.1328042Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1365959Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T12:13:56.1405495Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1452377Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T12:13:56.1491140Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1531055Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T12:13:56.1571400Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1618158Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T12:13:56.1656813Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1692632Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T12:13:56.1732145Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1768057Z Entering 'third_party/fbgemm/external/json' 2025-12-04T12:13:56.1806732Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1848568Z Entering 'third_party/flash-attention' 2025-12-04T12:13:56.1889288Z http.https://github.com/.extraheader 2025-12-04T12:13:56.1925809Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T12:13:56.1965752Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2010281Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T12:13:56.2049680Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2097875Z Entering 'third_party/flatbuffers' 2025-12-04T12:13:56.2139901Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2180038Z Entering 'third_party/fmt' 2025-12-04T12:13:56.2220609Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2256599Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T12:13:56.2297997Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2333406Z Entering 'third_party/gloo' 2025-12-04T12:13:56.2375405Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2410632Z Entering 'third_party/googletest' 2025-12-04T12:13:56.2451478Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2487404Z Entering 'third_party/ideep' 2025-12-04T12:13:56.2525751Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2563342Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T12:13:56.2602361Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2648102Z Entering 'third_party/ittapi' 2025-12-04T12:13:56.2687385Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2725160Z Entering 'third_party/kineto' 2025-12-04T12:13:56.2764820Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2802095Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T12:13:56.2840503Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2883508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T12:13:56.2922656Z http.https://github.com/.extraheader 2025-12-04T12:13:56.2962208Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T12:13:56.3003152Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3049771Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T12:13:56.3087033Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3128283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T12:13:56.3169346Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3205520Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T12:13:56.3244843Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3287698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T12:13:56.3325288Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3370921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T12:13:56.3409591Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3448928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T12:13:56.3487716Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3529429Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T12:13:56.3569485Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3612565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T12:13:56.3651333Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3687111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T12:13:56.3725657Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3766102Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T12:13:56.3805630Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3852392Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T12:13:56.3891604Z http.https://github.com/.extraheader 2025-12-04T12:13:56.3925967Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T12:13:56.3966761Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4012508Z Entering 'third_party/kleidiai' 2025-12-04T12:13:56.4052372Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4089084Z Entering 'third_party/mimalloc' 2025-12-04T12:13:56.4125779Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4162841Z Entering 'third_party/nlohmann' 2025-12-04T12:13:56.4202556Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4240734Z Entering 'third_party/onnx' 2025-12-04T12:13:56.4278803Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4335147Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T12:13:56.4375829Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4413844Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T12:13:56.4455601Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4492560Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T12:13:56.4529067Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4566587Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T12:13:56.4604237Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4644526Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T12:13:56.4683298Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4721061Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T12:13:56.4759833Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4798733Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T12:13:56.4836720Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4871985Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T12:13:56.4909301Z http.https://github.com/.extraheader 2025-12-04T12:13:56.4944052Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T12:13:56.4982399Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5018465Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T12:13:56.5057237Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5097761Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T12:13:56.5138494Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5176209Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T12:13:56.5215147Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5270954Z Entering 'third_party/pocketfft' 2025-12-04T12:13:56.5309376Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5345457Z Entering 'third_party/protobuf' 2025-12-04T12:13:56.5384114Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5425365Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T12:13:56.5463323Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5501442Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T12:13:56.5540671Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5578406Z Entering 'third_party/psimd' 2025-12-04T12:13:56.5618346Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5654047Z Entering 'third_party/pthreadpool' 2025-12-04T12:13:56.5693740Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5727574Z Entering 'third_party/pybind11' 2025-12-04T12:13:56.5767319Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5807233Z Entering 'third_party/python-peachpy' 2025-12-04T12:13:56.5846995Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5884804Z Entering 'third_party/sleef' 2025-12-04T12:13:56.5925546Z http.https://github.com/.extraheader 2025-12-04T12:13:56.5963459Z Entering 'third_party/tensorpipe' 2025-12-04T12:13:56.6003928Z http.https://github.com/.extraheader 2025-12-04T12:13:56.6040566Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T12:13:56.6079507Z http.https://github.com/.extraheader 2025-12-04T12:13:56.6113619Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T12:13:56.6152620Z http.https://github.com/.extraheader 2025-12-04T12:13:56.6189209Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T12:13:56.6225947Z http.https://github.com/.extraheader 2025-12-04T12:13:56.6263211Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T12:13:56.6305876Z http.https://github.com/.extraheader 2025-12-04T12:13:56.6342324Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T12:13:56.6381550Z http.https://github.com/.extraheader 2025-12-04T12:13:56.6440376Z [command]/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:56.6475181Z [command]/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url 2025-12-04T12:13:56.6803707Z Entering 'android/libs/fbjni' 2025-12-04T12:13:56.6829517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-12-04T12:13:56.6848170Z Entering 'third_party/FP16' 2025-12-04T12:13:56.6874365Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-12-04T12:13:56.6893081Z Entering 'third_party/FXdiv' 2025-12-04T12:13:56.6918441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-12-04T12:13:56.6939307Z Entering 'third_party/NNPACK' 2025-12-04T12:13:56.6966015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-12-04T12:13:56.6984174Z Entering 'third_party/NVTX' 2025-12-04T12:13:56.7011389Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-12-04T12:13:56.7029124Z Entering 'third_party/VulkanMemoryAllocator' 2025-12-04T12:13:56.7057178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-12-04T12:13:56.7074792Z Entering 'third_party/XNNPACK' 2025-12-04T12:13:56.7103078Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-12-04T12:13:56.7140242Z Entering 'third_party/aiter' 2025-12-04T12:13:56.7167519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-12-04T12:13:56.7186615Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-12-04T12:13:56.7212277Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-12-04T12:13:56.7238556Z Entering 'third_party/benchmark' 2025-12-04T12:13:56.7267119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-12-04T12:13:56.7287116Z Entering 'third_party/composable_kernel' 2025-12-04T12:13:56.7312492Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-12-04T12:13:56.7341592Z Entering 'third_party/cpp-httplib' 2025-12-04T12:13:56.7369327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-12-04T12:13:56.7387161Z Entering 'third_party/cpuinfo' 2025-12-04T12:13:56.7414795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-12-04T12:13:56.7431991Z Entering 'third_party/cudnn_frontend' 2025-12-04T12:13:56.7459560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-12-04T12:13:56.7475735Z Entering 'third_party/cutlass' 2025-12-04T12:13:56.7504367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-12-04T12:13:56.7532451Z Entering 'third_party/fbgemm' 2025-12-04T12:13:56.7559242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-12-04T12:13:56.7581277Z Entering 'third_party/fbgemm/external/asmjit' 2025-12-04T12:13:56.7607733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-12-04T12:13:56.7625251Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-12-04T12:13:56.7651897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-12-04T12:13:56.7676600Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-12-04T12:13:56.7703350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-12-04T12:13:56.7722850Z Entering 'third_party/fbgemm/external/cutlass' 2025-12-04T12:13:56.7749286Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-12-04T12:13:56.7777167Z Entering 'third_party/fbgemm/external/googletest' 2025-12-04T12:13:56.7803904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-12-04T12:13:56.7821764Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-12-04T12:13:56.7848068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-12-04T12:13:56.7865158Z Entering 'third_party/fbgemm/external/json' 2025-12-04T12:13:56.7890878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-12-04T12:13:56.7910559Z Entering 'third_party/flash-attention' 2025-12-04T12:13:56.7940419Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-12-04T12:13:56.7957133Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-12-04T12:13:56.7984635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-12-04T12:13:56.8010822Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-12-04T12:13:56.8034830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-12-04T12:13:56.8063429Z Entering 'third_party/flatbuffers' 2025-12-04T12:13:56.8090102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-12-04T12:13:56.8109831Z Entering 'third_party/fmt' 2025-12-04T12:13:56.8140025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-12-04T12:13:56.8156463Z Entering 'third_party/gemmlowp/gemmlowp' 2025-12-04T12:13:56.8185562Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-12-04T12:13:56.8204227Z Entering 'third_party/gloo' 2025-12-04T12:13:56.8230565Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-12-04T12:13:56.8249656Z Entering 'third_party/googletest' 2025-12-04T12:13:56.8274692Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-12-04T12:13:56.8294001Z Entering 'third_party/ideep' 2025-12-04T12:13:56.8320701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-12-04T12:13:56.8341289Z Entering 'third_party/ideep/mkl-dnn' 2025-12-04T12:13:56.8367687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-12-04T12:13:56.8391566Z Entering 'third_party/ittapi' 2025-12-04T12:13:56.8418972Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-12-04T12:13:56.8435526Z Entering 'third_party/kineto' 2025-12-04T12:13:56.8463301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-12-04T12:13:56.8483004Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-12-04T12:13:56.8509191Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-12-04T12:13:56.8525630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-12-04T12:13:56.8550997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-12-04T12:13:56.8571111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-12-04T12:13:56.8595639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-12-04T12:13:56.8614097Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-12-04T12:13:56.8638823Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-12-04T12:13:56.8659367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-12-04T12:13:56.8686228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-12-04T12:13:56.8703226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-12-04T12:13:56.8729014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-12-04T12:13:56.8749177Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-12-04T12:13:56.8776012Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-12-04T12:13:56.8791563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-12-04T12:13:56.8818787Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-12-04T12:13:56.8834915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-12-04T12:13:56.8862465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-12-04T12:13:56.8878862Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-12-04T12:13:56.8906325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-12-04T12:13:56.8924603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp' 2025-12-04T12:13:56.8949678Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T12:13:56.8966012Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T12:13:56.8990734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T12:13:56.9011444Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T12:13:56.9035821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T12:13:56.9059045Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-12-04T12:13:56.9086785Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-12-04T12:13:56.9104908Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-12-04T12:13:56.9129645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-12-04T12:13:56.9149073Z Entering 'third_party/kleidiai' 2025-12-04T12:13:56.9176742Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-12-04T12:13:56.9193438Z Entering 'third_party/mimalloc' 2025-12-04T12:13:56.9221131Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-12-04T12:13:56.9240975Z Entering 'third_party/nlohmann' 2025-12-04T12:13:56.9267839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-12-04T12:13:56.9288345Z Entering 'third_party/onnx' 2025-12-04T12:13:56.9313494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-12-04T12:13:56.9350366Z Entering 'third_party/onnx/third_party/pybind11' 2025-12-04T12:13:56.9378002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-12-04T12:13:56.9396074Z Entering 'third_party/opentelemetry-cpp' 2025-12-04T12:13:56.9424849Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-12-04T12:13:56.9444419Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-12-04T12:13:56.9469381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-12-04T12:13:56.9486984Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-12-04T12:13:56.9510417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-12-04T12:13:56.9526514Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-12-04T12:13:56.9551938Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-12-04T12:13:56.9569979Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-12-04T12:13:56.9593554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-12-04T12:13:56.9614138Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-12-04T12:13:56.9638613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-12-04T12:13:56.9657512Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-12-04T12:13:56.9683750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-12-04T12:13:56.9701054Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-12-04T12:13:56.9726956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-12-04T12:13:56.9743285Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-12-04T12:13:56.9769773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-12-04T12:13:56.9789230Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-12-04T12:13:56.9815822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-12-04T12:13:56.9833547Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-12-04T12:13:56.9860813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-12-04T12:13:56.9901925Z Entering 'third_party/pocketfft' 2025-12-04T12:13:56.9928342Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-12-04T12:13:56.9947056Z Entering 'third_party/protobuf' 2025-12-04T12:13:56.9974381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-12-04T12:13:56.9993845Z Entering 'third_party/protobuf/third_party/benchmark' 2025-12-04T12:13:57.0021864Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-12-04T12:13:57.0037468Z Entering 'third_party/protobuf/third_party/googletest' 2025-12-04T12:13:57.0064458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-12-04T12:13:57.0084863Z Entering 'third_party/psimd' 2025-12-04T12:13:57.0109830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-12-04T12:13:57.0126908Z Entering 'third_party/pthreadpool' 2025-12-04T12:13:57.0152801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-12-04T12:13:57.0172469Z Entering 'third_party/pybind11' 2025-12-04T12:13:57.0197998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-12-04T12:13:57.0217148Z Entering 'third_party/python-peachpy' 2025-12-04T12:13:57.0245685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-12-04T12:13:57.0263709Z Entering 'third_party/sleef' 2025-12-04T12:13:57.0291125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-12-04T12:13:57.0308815Z Entering 'third_party/tensorpipe' 2025-12-04T12:13:57.0335354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-12-04T12:13:57.0352419Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-12-04T12:13:57.0379484Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-12-04T12:13:57.0394652Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-12-04T12:13:57.0421503Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-12-04T12:13:57.0437000Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-12-04T12:13:57.0463693Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-12-04T12:13:57.0482816Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-12-04T12:13:57.0508003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-12-04T12:13:57.0525633Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-12-04T12:13:57.0549997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-12-04T12:13:57.0589063Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0617059Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0644675Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0670722Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0697654Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0725635Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0752086Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0776721Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0803803Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0829845Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0855484Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0882416Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0909358Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0935942Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0961464Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.0986827Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1012738Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1049514Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1063622Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1089928Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1115442Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1141834Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1167988Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1192341Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1216214Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1242865Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1267579Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1292763Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1320475Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1346023Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1370833Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1396154Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1420780Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1447738Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1473296Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1498034Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1524960Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1551281Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1575806Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1602589Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1629431Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1653747Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1680250Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1705503Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1731818Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1757011Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1784059Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1812041Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1838817Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1863972Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1890249Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1914716Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1940286Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1966257Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.1991293Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2015496Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2041508Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2066285Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2091610Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2116053Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2141149Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2166759Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2192400Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2217150Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2244962Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2270148Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2294675Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2319647Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2346565Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2373837Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2400274Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2426232Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2451744Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2476694Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2502065Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2527965Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2553906Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2578532Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2605834Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2631001Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2655938Z [command]/usr/bin/git config --file /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config --name-only --get-regexp ^includeIf\.gitdir: 2025-12-04T12:13:57.2769022Z A job completed hook has been configured by the self-hosted runner administrator 2025-12-04T12:13:57.2784595Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-12-04T12:13:57.2790461Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-12-04T12:13:57.2790921Z ##[endgroup] 2025-12-04T12:13:57.2876671Z [!ALERT!] Swap in detected! [!ALERT!] 2025-12-04T12:14:08.3325924Z [!ALERT!] Swap out detected [!ALERT!] 2025-12-04T12:14:26.9044084Z Cleaning up orphan processes